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ABSTRACT 


Will  a  machine  someday  be  held  accountable  for  its  actions?  This 
paper  provides  one  view  of  events  to  answer  the  question  “how 
trustworthy  and  reliable  must  an  Autonomous  Lethal  Engagement  (ALE) 
system  have  to  be  to  satisfy  ethical  and  legal  requirements?”  A  fictional 
legal  argument  acts  as  the  framework  to  debate  this  question. 

The  more  quantitative  side  of  reliability  relates  to  discrimination 
while  the  qualitative  aspect  of  trustworthiness  relates  to  proportionality. 
Context  matters  with  respect  to  the  ethical  argument.  Historically, 
acceptable  ethical  lines  at  both  the  national  and  societal  levels  vary 
based  on  the  context  surrounding  actions.  From  precedence,  how  one 
kills  matters  less  than  why  one  kills  with  two  notable  exceptions.  How 
one  kills  matters  with  regard  to  1)  distinction  of  target  and 
discrimination  of  friendly  or  neutral  entities  and  2)  the  effective 
proportionality  of  the  kill.  The  crux  of  the  central  thesis  question  is  a 
related  question  that  drives  the  theoretical  legal  proceeding;  “can  a 
machine  be  held  accountable  for  its  actions?”  By  taking  a  broad  look  at 
the  legal,  ethical,  philosophical,  and  technical  fields  surrounding  the 
emergence  of  ALEs,  one  potential  future  is  evaluated. 
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Introduction 


Although  our  intellect  always  longs  for  clarity  and  certainty, 
our  nature  often  finds  uncertainty  fascinating.  It  prefers  to 
day-dream  in  the  realms  of  change  and  luck  rather  than 
accompany  the  intellect  on  its  narrow  and  tortuous  path  of 
philosophical  enquiry  and  logical  deduction  only  to  arrive  - 
hardly  knowing  how  -  in  unfamiliar  surroundings  where  all 
the  usual  land-marks  seem  to  have  disappeared.  Unconfined 
hy  narrow  necessity,  it  can  revel  in  a  wealth  of  possibilities; 
which  inspire  courage  to  take  wing  and  dive  into  the  element 
of  daring  and  danger  like  a  fearless  swimmer  into  the  current. 

-  Carl  Von  Clausewitz 


Air  Force  Material  Command  (AFMC)  posed  the  following  research 
question:  “How  reliable  would  an  autonomous  lethal  engagement  system 
have  to  be  to  satisfy  ethical  and  legal  requirements?”1  The  lynch  pin  to 
this  question  is  legality,  more  specifically,  legal  accountability.  Assuming 
jurisdiction  of  all  parties,  the  legal  argument  is  reduced  to  one  of 
standing  and  accountability.  The  AFMC  point  of  contact  added  to  the 
question  with  the  following  statement:  “Before  we  invest  R&D  [research 
and  development]  dollars,  it  made  sense  to  me  to  think  through  some  of 
the  ethics  and  trust  issues  ...  You  might  also  note  that  I  proposed  this 
topic  from  an  Remotely  Piloted  Aircraft  (RPA)  perspective  but  you  could 
probably  apply  it  to  air  to  air  scenarios  as  well.”2 

What  follows  is  not  a  straightforward  answer  to  AFMC’s  question. 
However,  the  arguments  presented  should  indicate  the  non-uniqueness 
of  the  issue  relevant  to  reliability  and  ethics  while  focusing  on  the  unique 
aspects  of  autonomous  systems  related  to  trustworthiness  and  legality. 
Chapter  1  sets  the  stage  for  the  background  trial  as  an  underlying  story 
for  the  thesis.  Historical  case  studies  address  ethical  concerns  in 

1  Air  University  Research  Information  Management  System,  “AURIMS,  AY1 1,  Research  Topic  List 
(updated  22  Spe  10),”  Listed  on  line  296  as  Autonomous  Lethal  Engagements,  1 1AFMC004,  Submitted  by 
Air  Force  Material  Command. 

2  Kevin  Stamey,  Director  of  Engineering  AFPEO/ISR  &  SOF,  email  “Re:  Regarding  Autonomous  Lethal 
Engagements,”  (August  8,  201 1). 


Chapter  2.  Reliability,  related  to  the  acquisitions  process,  is  also 
addressed  in  Chapter  2.  Chapter  3  broaches  the  concept  of  trust. 
Although  trust  is  not  quantifiable,  societal  interpretations  of  social 
norms  and  transfers  of  moral  hazard  do  exert  influence.  Trust  from  one 
moral  agent  to  another  does  not  dissolve  accountability  from  the  trusting 
agent.  Accountability  must  exist  to  justify  or  explain  the  consequences  of 
failed  acts  of  trust.  While  legal  themes  run  the  course  of  the  thesis, 
Chapter  4  addresses  the  legal  concerns  of  accountability  and 
demonstrates  what  some  view  as  extreme,  yet  legal,  uses  of  autonomy 
related  to  discrimination. 

This  thesis  may  seem  fanciful.  Prior  to  dismissing  the  arguments 
surrounding  autonomous  lethal  systems,  the  reader  is  asked  to  examine 
other  significant  events  in  human  history  and  consider  how  fanciful 
discussion  surrounding  commonly  accepted  ideas  such  as  human  rights 
or  states  as  independent  actors  seemed  only  a  few  hundred  years  ago. 

In  1648  the  Peace  of  Westphalia  ended  the  Thirty  Years’  War.  The 
desire  of  the  signatories  to  ensure  the  treaty  lasted  beyond  the  life  span 
of  single  rulers  drove  the  concept  of  modern  statehood.  The  appointed 
representatives  of  the  various  royal  heads  of  state  signed  the 
international  treaty  under  authority  of  the  so-called  crowns,  a  continuing 
public  power,  and  not  the  personal  authority  of  the  individual  rulers. 
Straumann  argues  that  Westphalia  served  as  a  “historiography  of 
international  law  and  international  affairs”  and  is  thus  the  basis  for  the 
international  legal  norms  established  throughout  Europe.3  These  legal 
norms  became  the  basis  for  international  legal  sovereignty.  Prior  to  this 
act,  the  idea  of  statehood  existing  as  a  distinct  entity  apart  from 
individual  rulers  may  have  seemed  as  absurd  as  the  idea  of  legal 
standing  for  an  autonomous  machine. 


3  Benjamin  Straumann,  “The  Peace  of  Westphalia  as  a  Secular  Constitution,”  Journal  Constellations  15, 
no.  2  (2008),  http://www.iilj.org/aboutus/documents/straumann.westphalia.pdf  (accessed  April  23,  2012) 


Thoughts  of  science  fiction  may  cross  the  reader’s  mind  when  first 
encountering  the  assumptions  outlined  in  Chapter  1.  The  US  military,  in 
part  driven  by  efforts  to  reduce  personnel  liability  and  cut  operating 
costs,  is  moving  in  the  direction  of  more  autonomous  systems.  While 
discussing  possible  next  generation  RPAs,  Lt  General  Deptula  states, 

“The  third  major  attribute  of  the  MQ-X  (a  general  term  used  to  describe  a 
future  RPA)  will  be  autonomy,  since  it  is  likely  that  an  enemy  will  seek  to 
cut  the  links  between  the  operator  and  the  aircraft.  Today,  RPAs  know 
when  they  are  out  of  touch  with  their  operators,  and  if  that  happens, 
some  can  return  safely  to  base.  With  the  MQ-X,  the  Air  Force  wants  an 
aircraft  that  can  continue  the  mission  on  its  own,  if  control  links  are 
cut.”4  He  also  expanded  on  the  early  stages  of  autonomous 
discrimination;:  “The  things  that  we’re  working  very  hard  on  is 
automating  a  lot  of  what  analysts  do  individually.  And  so,  instead  of 
having  a  person  [watch]  a  video  screen  looking  for  a  person  to  come  out 
of  a  building,  I  can  come  up  with  the  technology  that  does  that 
automatically  ...  [and]  tracks  that  person  and  provides  notification.”5 

The  basic  assumption,  the  existence  of  machines  with  enough 
autonomy  to  execute  discriminatory  kill  criteria,  relies  on  technological 
advances  not  yet  realized.  Futurists,  such  as  Kurzweil,  proclaim  that 
these  events  will  occur  sooner  than  most  accept  as  possible  and  that  we 
will  likely  find  ourselves  beyond  the  precipice  before  we  realize  how  much 
life  has  changed. 


4  John  A.  Tirpak,  “The  RPA  Boom,”  The  Air  Force  Magzine.  Com,  August  2010.  http://  www.  airforce- 
magazine.  com /  MagazineArchive/Pages/2010/August%202010/0810RPA.aspx  (April  24,  2012). 

5  Tirpak,  “The  RPA  Boom.” 


Chapter  1:  A  Preliminary  Trial 


There  was  nothing  abnormally  interesting  about  the  January 
attacks.  The  Associate  Press  carried  the  story.  A  senior  terrorist  leader 
was  killed  as  part  of  ongoing  peacekeeping  operations.  The  sensational 
part  of  the  story  occurred  three  months  after  the  events. 

Also  killed  in  the  attack  were  several  civilians.  One  of  those  was  a 
twenty-two  year  old  college  student  traveling  in  the  region.  Parents  of  the 
student  were  outraged  at  the  loss  of  their  only  child  and  sought  justice. 
First,  the  parents  pressured  the  military.  Their  loss,  while  sad,  was 
acceptable  when  viewed  through  the  perspective  of  operational  necessity 
and  reasonable  proportionality.  The  senior  terrorist  leader  was 
responsible  for  the  deaths  of  thousands  of  innocent  civilians,  and  he  had 
been  hunted  for  years.  Collateral  damage  was  always  regrettable,  but  in 
this  case,  the  opportune  benefits  outweighed  the  unfortunate  costs. 
Stonewalled,  the  family  sought  answers  from  the  person  who  executed 
the  mission.  Could  this  soldier  not  have  waited  until  their  son  was  clear 
of  the  area  before  firing? 

When  the  parents  pushed  for  a  civil  case,  they  were  dumbfounded. 
The  person  responsible  simply  did  not  exist  -  at  least  not  in  a  legal 
sense.  Their  child  was  killed  through  the  actions  of  an  autonomous 
lethal  engagement  system.  The  family  knew  of  these  ALEs.  Their  prolific 
use  seeped  into  this  conflict  on  all  levels.  Ground,  sea,  and  air-based 
ALEs  existed.  If  they  thought  hard  about  it,  autonomous  systems  filled 
every  aspect  of  their  lives;  they  existed  in  all  forms  and  sizes.  Some  were 
human  sized.  Others  orbited  high  in  the  sky,  filled  with  waiting  fuel, 
serving  as  airborne  gas  stations.  Medically-based  autonomous  systems, 
designed  to  enter  the  body  and  cure  weakened  immune  systems  or 


strengthen  weakened  veins,  were  so  small  the  human  eye  could  not 
detect  their  existence.1 

The  family  was  at  a  loss  but  still  felt  the  need  to  point  at  someone 
and  ask  “why,  why  was  our  child  killed,  why  was  this  not  avoided,  what 
other  decisions  could  have  been  made?”  Out  of  desperation,  the  family 
charged  the  code  writers  of  the  ALE  for  proximate  cause  in  the  wrongful 
death  of  their  child.  Company  AutX  was  prepared. 


Premise  for  the  trial:  a  lack  of  causation  and  standing 

True,  the  company  had  procured  the  initial  code  that  fed  the 
genetic  algorithms  linked  into  a  neural  net.  The  algorithms  for  each  ALE 
evolved  independently  of  the  others  and  in  such  a  manner  that  the 
original  contextual  bounds  no  longer  existed.  Context  drives  decision. 
Killing  civilians  is  generally  an  unacceptable  act.  However,  it  is  accepted 
that  even  in  just  wars,  some  civilians  die  as  a  result  of  legitimate  military 
necessity.  Context  also  drives  definitions.  The  definition  of  a  civilian 
varies  from  war  to  war  based  on  the  political  context.  The  actions  of  the 
ALE,  as  influenced  by  unforeseeable  context,  were  themselves 
unforeseeable. 

Company  AutX  convinced  the  Court  of  the  lack  of  causation.  The 
decision  left  the  court  in  a  predicament.  By  allowing  one  finger  to  point 
away  from  the  company,  the  requirement  of  causation  pointed  another 
finger  at  a  machine.  That  machine  lacked  standing  in  the  court  of  law. 


John  A.  Tirpak,  “The  RPA  Boom,”  The  Air  Force  Magzine.  Com,  August  2010.  http://  www.  airforce- 
magazine.  com/  MagazineArchive/Pages/2010/August%202010/0810RPA.aspx  (April  24,  2012).  This 
section  refers  to  potential  future  advanced  autonomous  systems  though  semi-autonomous  systems  exit 
today  that  fill  similar  roles  as  those  described.  Tirpak  describes  some  of  these  systems  as  follows:  “The 
UAS  Flight  Plan  does  not  limit  itself  to  items  in  the  Predator-Reaper  or  Global  Hawk  classes.  At  the  low 
end  of  the  spectrum,  the  Air  Force  is  busy  developing  ‘nano-micro’  systems,  of  the  size  of  a  small  bird  or 
insect,  that  will  be  able  to  penetrate  rooms  to  conduct  reconnaissance,  cyber  attack,  or  even  lethal 
operations.  The  service  already  fields  a  number  of  hand-launched  ‘man-portable’  systems  such  as  the 
Raven,  which  can  look  over  hills  and  resembles  toy  radio-controlled  aircraft,  and  is  investigating  air- 
launched  systems  that  may  or  may  not  be  expendable.  At  the  high  end,  the  roadmap  talks  about  ‘tanker¬ 
sized’  RPAs  that  could  perform  long,  dull  jobs  such  as  serving  as  communications  nodes  while 
simultaneously  providing  air  refueling  and  GMTI  missions  with  onboard  radars.” 


Without  legal  standing,  the  ALE  in  question  is  no  more  than  defective 
property.  Destruction  of  the  defective  property  was  one  simple  solution  to 
the  issue.  However,  the  statements  from  Company  AutX  intrigued  the 
court.  Was  this  ALE  simply  defective  property  or  something  more?  The 
court  ordered  a  legal  proceeding  and  appointed  counsel  to  the  ALE. 

Determination  of  the  machine’s  legal  standing  is  the  basis  for  the 
following  fictitious  legal  proceeding.  Accountability  of  causation  requires 
legal  standing.  The  court  issued  an  injunctive  order  to  spare  the 
existence  of  this  ALE  pending  a  possible  declarative  judgment.  A 
declarative  judgment  stating  the  ALE  was  a  person,  or  some  other  entity 
with  legal  standing,  would  force  a  causation  trial.  The  causation  trial 
would  determine  the  guilt  or  innocence  of  the  ALE.  For  now,  the  trial 
focused  on  the  question  “can  a  machine  be  held  accountable  for 
actions?” 

The  media  flocked  to  this  trial.  A  machine,  demanding  recognition 
of  unalienable  rights,  the  sheer  recognition  of  which  could  place  the  ALE 
in  greater  peril  than  if  it  were  considered  defective  property.  Defective 
property  can  be  corrected  or  scrapped,  depending  on  the  cost. 

Completely  scrapping  a  machine  as  complex  as  the  ALE  did  not  make 
sense.  Ridding  the  machine  of  any  distinguishing  features  developed  over 
time  would  destroy  the  essence  of  the  machine.  Death  or  a  complete 
personality  and  memory  lobotomy  are  the  only  human  comparisons  for 
such  a  process. 

Replays  of  his  college  philosophy  debates  ran  through  Judge 
Minos’s  head  as  the  bailiff  called  the  room  to  order.  Honorable  Minos 
observed  the  normality  of  his  courtroom  with  an  odd  clarity.  The  room 
was  occupied,  as  it  would  be  on  a  nominal  morning.  The  roles  and 
players  were  well  defined  with  the  exception  of  the  defendant.  Judge 
Minos  stared  at  the  ALE  and  thought  to  himself,  ‘here  we  start  to  define 
who  or  what  you  are  and  what  responsibilities  are  owed  between  you  and 
the  society  that  created  you.’ 


A  side  bar  for  definitions 

The  term  autonomous  has  widely  varying  definitions.  Unless 
qualified  within  the  text  or  footnotes  the  following  definitions  apply. 

Autonomous :  A  robot  “capable  of  making  at  least  some  of  the  major 
decisions  about  their  actions  using  their  own  programming.”2  This 
definition  covers  a  robot  vacuum  cleaner  deciding  on  a  vacuum  pattern 
to  the  moral  and  ethical  reasoning  of  a  robotic  caregiver  deciding  the  best 
way  to  interact  with  a  patient. 


Telerobot  Per  Sullins,  a  telerobot  is  a  “remotely  controlled  machines  that 
makes  only  minimal  autonomous  decisions.”  3  The  operator  provides  the 
intelligence  for  the  machine  such  as  the  NASA  Mars  Rovers,  telerobotic 
surgery,  telerobotic  nurses,  Predator  drones,  and  the  Army  SWORD. 


Unmanned  Aerial  Vehicle  (UAV):  From  the  2008  Dictionary  of  Military  and 
Associate  Terms,  “Powered  aerial  vehicle  that  does  not  carry  a  human 
operator,  . . .  can  fly  autonomously  . . . ,  and  can  carry  a  lethal  or  non-lethal 
payload.”4  The  USAF  has  replaced  the  term  UAV  with  Remotely  Piloted 


2  John  P.  Sullins,  “When  is  a  Robot  a  Moral  Agent?”  International  Review  of  Information  Ethics  6, 
(December  2006):  25-26.  To  bypass  a  “full  discussion  of  the  meaning  of  ‘autonomy’”  as  a  philosophical 
subject,  Sullins  uses  the  roboticists  definition:  “autonomous  robots  must  be  capable  of  making  at  least  some 
of  the  major  decisions  about  their  actions  using  their  own  programing.”  This  definition  covers  a  robot 
vacuum  cleaner  deciding  on  a  vacuum  pattern  to  the  moral  and  ethical  reasoning  of  a  robotic  caregiver 
deciding  the  best  way  to  interact  with  a  patient. 

3  Sullins,  “When  is  a  Robot  a  Moral  Agent?”  25-26.  Sullins  defines  a  telerobot  as  “remotely  controlled 
machines  that  make  only  minimal  autonomous  decisions.”  The  operator  provides  the  intelligence  for  the 
machine  such  as  the  NASA  Mars  Rovers,  telerobotic  surgery,  telerobotic  nurses,  Predator  drones,  and  the 
Army  SWORD. 

4  Joint  Publication  1-02,  Department  of  Defense  Dictionary  of  Military  and  Associated  Terms,  April  12, 
2001  (amended  October  17,  2008),  579.  www.dtic.mil/  doctrine/  new_pubs/ jpl_02.pdfSimilar  (accessed 
January  2012).  Chris  Jenks  expanded  on  this  definition  with  his  interpretation  of  a  GAO  report  in  his 
opinion  titled  Law  From  Above:  Unmanned  Aerial  systems,  Use  of  Force,  and  the  Law  of Armed  Conflict. 


Aircraft  (RPA)  to  highlight  the  aircraft  dependence  on  manned  crew 
interaction.* * * * 5 

Additional  definitions  are  located  in  Appendix  B:  Definitions. 


How  did  we  get  here? 

Judge  Minos  listened  intently  to  both  sides  of  the  initial  argument. 
He  felt  the  intensity  of  the  public  light  shining  on  his  courtroom  and 
knew  his  ruling  would  cast  a  long  shadow.  It  would  affect  far  more  than 
the  fate  of  the  ALE  standing  in  judgment  before  him.  His  decision  would 
set  the  precedent  for  defining  personhood  while  opening  the  door  for 
sentience,  free  will,  and  the  recognition  of  legal  rights  for  what  many 
viewed  as  an  elaborate  calculator.  Before  proceeding  farther  in  the  trial, 
Judge  Minos  needed  to  understand  how  this  point  in  time  came  to  be. 

Stepping  back  from  the  trial  the  judge  realized  there  were  two 
arguments  he  desired  to  hear.  He  wanted  to  understand  how  humans 
have  justified  the  use  of  and  allowed  a  so-called  thinking  machine  to  do 
their  killing.  Throughout  his  life,  the  Judge  had  witnessed  unthinkable 
human  atrocities  and  on  the  rarest  occasions,  extraordinary  kindness. 
He  had  always  favored  the  gray  lines  when  examining  human  kind. 
Humans  span  the  length  of  good  and  evil  and  he  believed  the  worst 
atrocities  committed  by  and  to  the  human  race  were  executed  by  those 
who  believed  they  were  in  the  right.  If  this  same  logic  holds  for  the 


Jenks  states  in  his  footnote  on  page  653  that  “the  distinction  between  autonomously  and  remotely  piloted 
UAVs  is  that  ‘[a]n  autonomously  piloted  UAV  is  one  that  is  pre-programmed  for  its  mission  before  it  takes 

off.  It  then  flies  its  mission  without  a  ground-based  pilot.  A  remotely  piloted  UAV  is  controlled  by  a  pilot 

in  a  control  station  on  the  ground  during  the  flight.”  GAO  report,  supra  note  3,  at  4  n.5.”’  The  term 

autonomous  used  to  describe  the  ALE  throughout  this  thesis  exceeds  the  autonomy  of  a  machine  “that  is 
pre-programmed  for  its  mission  before  it  takes  off.”  The  autonomous  nature  of  ALEs  reflect  a  capability  to 
make  major  decisions. 

5  Tirpak,  “The  RPA  Boom.”  Tirpak,  while  interviewing  Lt.  Gen.  David  A.  Deptula,  Air  Force  deputy  chief 
of  staff  for  intelligence-surveillance-reconnaissance  (ISR)  provided  insight  to  USAF  decision  to  drop  the 
term  UAV  and  switch  to  RPA:  “Since  the  Flight  Plan  was  published  in  mid-2009,  the  Air  Force  has 
dropped  the  term  ‘unmanned  aircraft  systems’  because  ‘there’s  nothing  unmanned  about  them,’  Deptula 
said.  It  can  take  as  many  as  170  persons  to  launch,  fly,  and  maintain  such  an  aircraft  as  well  as  to  process 
and  disseminate  its  ISR  products.  The  UAS  term  gave  the  false  impression  that  they  required  little 
manpower  investment,  Deptula  noted. 


creation  of  the  ALE  sitting  in  front  of  the  Judge,  how  did  we  come  to 
justify  the  ethical  use  of  such  a  machine? 

The  Judge  called  for  a  recess  so  that  he  could  further  educate 
himself  on  the  issue  and  research  his  own  question.  As  he  rose  to  leave 
the  room,  the  thought  occurred  to  him  that  society  may  not  have  chosen 
to  justify  the  ethical  use  of  the  ALEs.  The  judge  began  to  question  if 
society  as  a  whole  was  even  aware  of  how  far  the  autonomous  systems 
penetrated  daily  lives.  Was  society’s  path  predetermined  by  the  growth  of 
technology  or  did  society  intent  and  individual  responsibility  construct 
the  situation? 


Chapter  2:  Ethical  Justification  Using  Historical  Case  Studies 


War  transforms  the  social  and  political  orders  in  which  we 
live,  just  as  it  obliterates  our  precious  certainties. 

-  Barkawi  and  Brighton, 

The  Scientific  Way  of  Warfare 


The  escalating  use  of  machines  in  war  has  spawned  increased 
attention  on  concepts  of  justness  in  war.  Separating  out  this  aspect  from 
the  broader  Just  War  debate  and  relegating  it  to  a  section  dealing  with 
changing  societal  norms  allows  distinctive  focus  on  the  historically- 
backed  sliding  scale  of  justified  ethics  in  war. 1 

Historical  case  studies  support  the  concept  of  sliding  justified 
ethical  behavior.  Accepting  the  realities  of  a  sliding  justified  scale 
provides  the  acquisitions  corps  and  operational  planners  with  resource 
boundaries.  It  also  allows  operational  commanders  a  measure  to  quantify 
which  weapons  are  not  acceptable  given  their  potential  use  allowing  for 
extant  or  probable  political  or  sociological  circumstances. 

The  defense  acquisitions  field  uses  reliability  and  confidence  values 
to  provide  predictions  for  success  rates  of  weapons.2  Establishing  the 
reliability  of  an  autonomous  system  could  follow  a  similar  path  to  the 
precedence  of  success  rates  used  for  weapon  systems.  Ethical  use  of  a 
system  implies  an  aspect  of  control.  Reliability  and  success  rates 


1  For  an  overview  of  the  Just  War  debate  see  Michael  Walzer,  Just  and  Unjust  Wars,  4th  Ed  (New  York 
NY:  Basic  Books,  2006),  focus  on  chapter  5  of  Everett  Carl  Dolman,  Pure  Strategy:  Power  and  Principle 
in  the  Space  and  Information  Age  (New  York  NY:  Frank  Cass,  2005),  Robert  B.  Strassler,  ed.,  The 
Landmark  Thucydides  (New  York  NY:  The  Free  Press,  1996),  Edward  D.  Mansfield  and  Jack  Snyder, 
Electing  to  Fight  (Cambridge  MA:  MIT  Press,  2007),  Peter  Ackerman  and  Jack  Duvall,  A  Force  More 
Powerful  (New  York  NY,  Palgrave,  2000),  US  Bishops,  The  Challenge  of  Peace,  May  3,  1983, 
http://old.usccb.org/  sdwp  /international/  TheChallengeofPeace.pdf  18-22  (accessed  May  9,  2012). 

2  Office  of  the  Undersecretary  of  Defense  for  Acquisitions,  Technology,  and  Eogistics,  Report  of  the 
Defense  Science  Board  Task  Force  on  Developmental  Test  and  Evaluation  (Washington,  DC:  Government 
Printing  Office,  May  2008)  http://  www.  acq.  osd.  mil /  dsb/  reports/ AD A482504.pdf  (accessed  on  March 
28,  2012)  21-22.  From  the  1970s  to  the  mid  1990s  the  defense  services  executed  a  concerted  effort  to 
employ  reliability  standards  and  testing  from  the  design  phase  through  fielding.  Reliability  programs 
remain  standard  on  aircraft  programs  in  both  the  Air  Force  and  Navy.  These  programs  have  reduced  the 
mean  time  between  failure  rates  as  well  as  achieving  designed  suitability  of  systems. 


demonstrate  aspects  of  control  and  predictability.  Trust  and  fear  are 
similar  in  that  both  are  individual  and  social  aspects  of  non-quantifiable 
reactions. 

A  socially  constructed  fear  of  autonomous  systems  may  drive  a 
technology  gap  that  places  the  US  at  risk  of  falling  behind  emerging 
military  powers.  An  analytical  evaluation  of  reliability  should  drive  the 
discriminatory  side  of  the  argument  while  qualitative  social  norms  and 
an  ethically  motivated  evaluation  of  context  introduce  trustworthy 
concepts. 


The  history  of  war  affords  little  hope  that 
nations  which  are  fighting  for  their  lives 
and  beliefs  will  be  restricted  in  their 
conduct  of  the  war  by  moral  factors. 

-  Arthur  W.  Tedder 


Sliding  scale  of  justification 

Justification  related  to  war  falls  to  two  large  categories.  The 
general  category  of  jus  ad  bellum  relates  to  proper  cause  and 
circumstances  for  going  to  war.  Specific  conflict  justifications  such  as  the 
Law  of  Armed  Conflict  or  regional  Rules  of  Engagement  fall  under  jus  en 
bello,  or  justice  within  war.  Discrimination  and  proportionality  relate  to 
varying  degrees  of  jus  ad  bellum  and  jus  en  bello.  The  following  historical 
examples  highlight  a  sliding  scale  of  justice  in  war  related  through  an 
examination  of  varying  degrees  of  discrimination  and  proportionality. 

Historical  case  studies  demonstrate  a  sliding  scale  of  ethical 
behavior  due  to  surrounding  circumstances.  The  fire  and  carpet-bombing 
efforts  in  both  theaters  of  World  War  II  express  how  the  nature  of  the 
conflict,  as  well  as  acceptable  technological  limitations,  resulted  in  wider 
discriminatory  efforts  than  those  of  today’s  politically  charged  precision 


munitions  era.3  World  War  II  also  demonstrated  a  sliding  scale  of 
acceptable  proportionality.  The  military  utility  gained  from  releasing  two 
mostly  untested  atomic  weapons  on  civilian  populations  reveals  a 
willingness  to  accept  high  damage  to  civilian  persons  and  property  in  an 
effort  to  gain  military  advantage.4 


Hit  oil  if  visual  assured;  otherwise,  Berlin  -  center  of  city. 

Gen  Carl  A.  Spaatz’s  order 
to  Lt  Gen  James  Doolittle 
February  1,  1945 


Addressing  discrimination  and  proportionality 

Case  1 :  World  War  II  Fire,  Carpet,  and  Atomic  Bombing 

Carpet-bombing,  fire-bombing,  and  the  release  of  atomic  weapons 
over  civilian  populations  in  Japan  highlight  the  indiscriminate  nature  of 
WWII  aerial  bombing,  but  the  decisions  to  do  so  were  part  of  an  interplay 
of  technological  and  socio-political  circumstances.  The  evolution  of  air 
power  in  WWI  moved  from  primarily  reconnaissance  to  interdiction.  In 
his  1925  Winged  Defense,  Billy  Mitchell  offered  a  strategic  view  of  air 
power  that  included  direct  bombing  of  civilian  areas  used  to  support  the 
military  industry.  His  interpretation  of  a  city  as  a  military  target  and  the 
strategic  implications  from  bombing  are  supported  with  his  statements, 
“For  attacking  cities  that  are  producing  great  quantities  of  war  munitions 
that  are  necessary  for  the  maintenance  of  an  enemy  army  and  country  in 
case  of  war,  the  air  force  offers  and  entirely  new  method  of  subduing 
them.  Now  an  attack  from  the  air  force  using  explosive  bombs  and  gas 


3  For  a  review  of  the  strategic  bombing  campaign  of  World  War  II  see  the  following:  Tami  Davis  Biddle, 
Rhetoric  and  Reality  in  Air  Warfare  (Princeton  NJ:  Princeton  University  Press,  2002), Richard  J  Overy,  The 
Air  War  (Washington  DC,  Potomac  Books,  2005),  Michael  S  Sherry,  The  Rise  of  American  Air  Power 
(New  Haven  CT,  The  Yale  University  Press,  1987),  Randall  T  Wakelam,  The  Science  of  Bombing: 
Operational  Reseach  in  RAF  Bomber  Command  (Buffalo  NY,  The  University  of  Toronto  Press,  2009), 
Thomas  E  Griffith,  Jr,  MacArthur’s  Airman  (Lawrence  KA:  University  Press  of  Kansas,  1998),  as  well  as 
the  books  referenced  in  section  2.1. 

4  Lawrence  Freedman,  The  evolution  of  Nuclear  Strategy,  third  edition  (New  York  NY,  Palgrave 
MacMillian,  2003),  4-12,  16. 


may  cause  the  complete  evacuation  of  and  cessation  of  industry  in  these 
places.  This  would  deprive  armies,  air  forces,  and  navies  even,  of  their 
means  of  maintenance.”5  Mitchell’s  proposal  met  the  first  half  of  the 
discrimination  criteria  in  just  war  theory  -  prosecution  of  military 
targets.  His  writings  indicated  that  the  “dominate  theme  ...  was  not  the 
desire  to  attack  civilians  directly,  but  rather  the  desire  to  sever  the 
populace  from  the  sources  of  production...  Achieving  that  goal  might 
cause  some  civilian  deaths,  but  the  number  would  pale  compared  to  the 
deaths  produced  by  ground  war  between  industrialized  powers.”6 

As  the  war  stretched  on,  “the  impetus  to  end  the  war  quickly 
provided  few  limits  to  the  definition  of  ‘military  objective.’”7  The  American 
ethical  line  of  discrimination  shifted  with  alibis  such  as  the  need  to 
hasten  “the  war’s  end  and  saving  American  lives  in  the  process.”8  A 
desire  to  inflict  great  amounts  of  pain  to  compel  the  Axis  powers  into 
submission  replaced  discrimination  requirements  for  both  identification 
and  targeting. 

The  Geneva  Conference  of  1932-1934,  the  1945  Charter  of  the 
International  Military  Tribunal,  and  the  1949  Geneva  Convention  are 
examples  of  “immediate  post-war  declarations  and  conventions 
constitute [ing]  a  retrospective  indictment  of  the  practices  they 
outlawed.”9  These  conventions  and  documents  support  the  notion  that 
the  Allied  governments  understood  the  actions  directed  towards  civilian 
targeting  were  wrong  but  still  chose  to  act  to  counter  a  greater  evil. 10 

Allied  Air  Forces  treated  civilian  populations  as  a  military  object. 
Justification  for  bombing  of  cities  ranged  from  a  strategy  of  breaking  the 

5  William  “Billy”  Mitchell,  Winged  Defense:  The  development  and  possibilities  of  modern  air  power - 
economic  and  military  (Tuscaloosa  AL:  The  University  of  Alabama  Press,  2009),  5-6. 

6  Mark  Clodfelter,  Beneficial  Bombing,  42. 

7  Mark  Clodfelter,  Beneficial  Bombing,  151. 

8  Mark  Clodfelter,  Beneficial  Bombing,  1 86.  (original  emphasis) 

9  A.C.  Grayling  Among  the  Dead  Cities:  The  history  and  moral  legacy  of  the  WWII  bombing  of  civilians  in 
Germany  and  Japan  (New  York  NY:  Walker  and  Company,  2006). 

10  Both  Grayling  and  Clodfelter  imply  support  to  the  opinion  that  the  conferences  acknowledged  a  moral 
relative  bad  committed  by  the  Allies  but  justified  as  an  absolute  good. 


enemy  economically,  logistically,  or  by  breaking  the  enemy’s  will  to  fight 
by  destroying  the  civilian  will  to  support  the  war.  The  fire-bombing  of 
Tokyo  and  four  other  of  Japan’s  largest  cites  as  well  as  the  “pounding  of 
Berlin  and  Dresden  ...  demonstrated  a  willingness  to  target  civilians 
directly  rather  than  relying  on  complementary  pain  caused  by  targeting 
nearby  government  offices  (Berlin)  or  rail  yards  (Dresden).  In  the 
Japanese  case,  ‘areas  assigned  were  selected  on  the  basis  of  a 
compromise  between  industrial  importance  and  susceptibility  to  fire.’ ... 
Their  intent  was  to  kill  people  and  destroy  homes,  which  would  indirectly 
affect  industrial  production.”11 

Discrimination  and  proportionality  are  linked  through  intent  and 
result.  A  discriminate  strike  is  one  that  not  only  identifies  a  target  but 
also  one  that  intends  a  proportionate  effect.  Actual  proportionality  is 
measured  by  the  result  regardless  of  intent. 12  The  intent  of  fire  bombing 
and  the  release  of  atomic  weapons  over  Japan  was  some  combination  of 
destroying  military  manufacturing,  breaking  civilian  will,  and 
punishment.  The  bombing  resulted  in  one  hundred  thousand  civilian 
deaths  in  one  night  alone.13 


Case  2:  Kosovo:  an  increased  reliance  on  precision  weapons 


11  Mark  Clodfelter,  Beneficial  Bombing,,  188-189.  Additiaonal  support  is  found  on  page  149  where 
Clodfelter  recounts  a  press  conference  held  by  RAF  Air  Commodore  C.M.  Grierson  in  February  1945. 
Grierson  “stated  that  bombing  population  centers  caused  the  Germans  difficulty  because  it  forced  them  to 
send  in  trains  carrying  relief  supplies  and  set  out  trains  carrying  homeless  civilians.” 

12  Chris  Jenks  and  Geoffrey  Corn,  “Siren  Song:  The  Implications  of  the  Goldstone  Report  on  International 
Criminal  Law.”  http//ssrn. com/abstract/ 1788542  (accessed  January  2012).  Jenks  and  Corn  state  on  page  4, 
that  an  objective  standard  of  assessment  is  used  to  evaluate  discriminatory  and  proportional  acts  through 
the  “subjective  perspective  of  the  military  commander  whose  actions  are  at  issue  and  based  on  the 
information  available  at  the  time  the  decision  was  made.”  Additionally,  Jenks  and  Corn  argue  on  page  6, 
“commanders  are  not  held  liable  based  on  retrospective  assessment  of  facts  and  circumstances,  often 
referred  to  as  the  ‘Rendulic  Rule.’”  Lothar  Rendulic  was  a  WWII  German  General  accused  of 
indiscriminately  and  disproportionally  ordering  a  ‘scorched  earth’  policy  to  slow  the  advancing  Russian 
forces. 

13  Joseph  Coleman,  “1945  Tokyo  Firebombing  left  Legacy  of  Terror,  Pain,”  Associated  Press,  Common 
Dreams ,  March  10,2005.  http://www.commondreams.org/headlines05/0310-08.htm  (accessed  on  April  1, 
2012)  On  the  night  between  March  9th  and  10th,  300  B-29s  dropped  an  estimated  half  million  M69 
incendiary  cylinders  over  16  square  miles  of  Tokyo.  “The  official  death  toll  was  some  83,000,  but 
historians  generally  agree  that  victims  unaccounted  for  bring  the  figure  to  around  1 00,000  overwhelmingly 
civilians.  It  is  widely  considered  to  be  the  most  devastating  air  raid  in  history.” 


On  March  24th,  1999,  NATO  engaged  the  Federal  Republic  of 
Yugoslavia  in  war.  Operation  Allied  Force  (OAF)  was  motivated  by  three 
primary  interests:  “ensuring  the  stability  of  Eastern  Europe;  thwarting 
ethnic  cleansings;  and  ensuring  NATO’s  credibility.”14  OAF  was  the  first 
time  NATO  used  military  action  against  a  state  for  humanitarian  reasons 
within  a  sovereign  nation;  essentially,  it  was  NATO’s  first  war.  OAF  was  a 
“politically  constrained  coercive  diplomacy”  effort. 15  The  first  phase  of  the 
operation,  Phased  Air  Operations  Plan,  was  a  limited  air  operation 
designed  to  gradually  increase  pressure  on  then-Serbian  President 
Milosevic.  Politically  constrained  discrimination,  through  target 
identification,  restricted  the  type  and  amount  of  targets  to  a  list  that 
lasted  only  three  days.  By  the  end  of  the  first  week,  after  all  authorized 
targets  had  been  hit  at  least  once,  General  Ralston  drafted  four  general 
target  procedural  rules  to  widen  the  target  identification.  Three  of  the 
four  rules  were  centered  on  discrimination  while  the  fourth  limited 
proportionality. 16 

NATO  aircraft  were  restricted  to  an  altitude  band  of  15,000  - 
20,000  feet  in  the  initial  phase  of  OAF.  While  this  altitude  band  co¬ 
aligned  with  several  Surface-to-Air  engagement  altitudes,  the  band  was 
selected  to  allow  for  optimal  Laser  Guided  Bomb  attacks.17  Attack 
missions  were  called  off  or  reassigned  if  civilians  were  located  within  the 
strike  area.  NATO  aircraft  assumed  greater  risk  to  “trash  fire”  over 
Kosovo  as  the  rate  of  inadvertent  civilian  casualties  increased. 18  NATO 
aircraft  descended  down  to  5,000  feet  for  identification  and  to  8,000  feet 


14  Dag  Henriksen,  NATO’s  Gamble:  Combining  Diplomacy  Airpower  in  the  Kosovo  Crisis,  1998-1999 
(Annapolis:  Naval  Institute  Press,  2007),  7. 

15  Henriksen,  NATO’s  Gamble,  87. 

16  Henriksen,  NATO’s  Gamble,  20-22.  Discrimination  focuses  on  identifying  and  minimizing  an  attack  to 
enemy  targets.  A  lack  of  discrimination  or  perceived  discrimination  can  counter  legitimate  political  aims. 
Henriksen  lists  General  Ralston’s  four  rules  as:  1)  don’t  bomb  Montenegro  unless  under  threat,  2)  don’t 
bomb  within  5  miles  of  downtown  Belgrade  without  additional  NATO  approval,  3)  unintended  casualties 
must  be  well  within  proportional  expectations  as  set  by  NATO,  4)  don’t  bomb  the  electric  grid. 

17  Benjamin  Lambeth,  NATO’s  air  war  for  Kosovo:  a  strategic  and  operational  assessment,  (Washington, 
DC:  RAND  Corporation,  2001),  140-142. 

18  Lambeth,  NATO’s  air  war  for  Kosovo,  141. 


to  release  non-precision  guided  munitions  in  an  effort  to  increase 
discrimination  and  precision. 19  Kosovo  is  an  example  of  political  will 
tipping  the  line  of  ethical  justification  for  the  use  of  military  force 
towards  prudence  to  the  point  of  limiting  a  decisive  blow  in  favor  of 
progressive  action.  The  international  political  pressure  surrounding 
Kosovo  tipped  the  proportionate  military  response  away  from  action 
required  to  meet  military  necessity,  such  as  ceasing  genocide,  in  favor  of 
maintaining  the  alliance  and  legitimacy  of  NATO. 


What  shall  it  profit  us  if  we  win  the  victory  and  lose  our 
civilization?  I  am  sure  we  must  be  far  more  selective  in  the 
allocation  of  our  national  effort  to  military  defence.  We  must 
pay  far  more  attention  to  the  principle  of  economy  of  force  - 
and  when  I  say  ‘economy’  I  do  not  mean  the  false  economy 
got  by  doing  things  on  the  cheap,  but  the  economy  which 
results  from  keeping  every  component  part  of  the  national  war 
machine  properly  balanced  in  relation  to  the  rest  of  the 
machine  -  the  economy,  moreover,  which  organises  the  armed 
forces  for  speed  and  quick  decision. 

-  Arthur  W.  Tedder 


Acquisitions 

As  stated  previously,  accepting  the  realities  of  a  sliding  justified 
scale  provides  the  acquisitions  corps  and  operational  planners  with 
resource  boundaries.  It  also  allows  operational  commanders  a  means  to 
quantify  which  weapons  are  not  acceptable  given  the  political  or 
sociological  circumstances.  Additional  research  in  this  area  could  provide 
variables  to  assess  the  discriminatory  and  proportional  affects  of  systems 
in  a  similar  way  that  reliability  and  confidence  values  provide  a 
prediction  for  success  rate. 

Reliability  and  confidence  values  communicate  expected 
performance  of  acquisition  systems.  Specifically,  “reliability  is  a  measure 


19  Lambeth,  NA  TO’s  air  war  for  Kosovo,  1 42. 


of  how  well  a  product  will  perform  under  a  certain  set  of  conditions  for  a 
specified  amount  of  time.  Many  times,  reliability  will  be  stated  along  with 
a  minimum  confidence  level.  Reliability  and  confidence  are  two  separate 
concepts.  Reliability  refers  to  a  failure  rate,  while  confidence  refers  to  the 
minimum  certainty  that  the  claimed  failure  rate  is  accurate.”20  The 
numbers  are  tested,  verified  and  used  to  support  doctrine  development. 
Operators  are  concerned  with  the  success  rate  of  a  system  under  given 
circumstance.  The  success  rate  of  a  system  is  the  reliability  for  a  given 
probability  or  confidence  value.21 

Reliability  is  quantifiable  while  trust  is  a  qualified  property.  The 
quantifiable  nature  of  reliability  supports  discrimination  while  the 
qualified  nature  of  trust  supports  proportionality.  Trustworthiness,  as  a 
characteristic,  is  more  difficult  to  stretch  the  definition  from  one  of 
repeatability  and  expected  results  to  one  of  desired  results. 
Proportionality  is  a  qualitative  assessment  of  the  real  and  likely 
consequences  balanced  against  military  necessity.  This  is  where  the 
trustworthy  nature  of  the  ALE  comes  into  question.  While  rarely  specified 
quantitatively,  a  minimum  level  of  discrimination  and  proportionality  is 
often  communicated  through  general  orders.  Justifying  the  use  of  lethal 
autonomous  systems  requires  acknowledging  an  acceptable  level  of  both 
reliability  and  trust. 

For  example,  munitions  are  often  delivered  to  the  field  with 
Circular  Error  Probable  (CEP)  values  that  serves  as  “an  indicator  of  the 
delivery  accuracy  of  a  weapon  system,  used  as  a  factor  in  determining 
probable  damage  to  a  target.  It  is  the  radius  of  a  circle  within  which  half 
of  a  missile's  projectiles  are  expected  to  fall.”22  Military  members 
operating  within  the  bounds  of  the  Law  of  Armed  Conflict,  require  an 


20  Maximator  Test,  LLC,  “Reliability,”  white  paper,  http://www.maximator-test.com/about/White-Paper- 
Reliability.pdf  (accessed  March  27,  2012). 

21  Quality  Portal  references,  http://thequalityportal.com/glossary/rc.pdf  (accessed  January  2012). 

22  Farlex,  The  Free  Dictionary,  http://www.thefreedictionary.com/  circular+  error+  probable  (accessed  on 
March  28  2012). 


ability  to  discriminate  and  execute  within  proportional  response  levels. 
An  autonomous  system  executing  on  behalf  of  the  nation  must,  at  a 
minimum,  meet  these  same  standards.  Just  as  inexperienced  or  raw 
recruits  should  not  be  placed  in  positions  to  make  decisions  regarding 
discrimination  and  proportionality  above  their  pay  level,  an  autonomous 
system  should  not  be  used  beyond  its  capabilities.  Design  characteristics 
defined  during  the  acquisition  process  can  bound  the  minimum 
capacities  of  future  systems.  This  minimum  threshold  ensures  the  basic 
user  needs  are  met  as  well  as  ensuring  the  acquisition  process  efficiently 
uses  resources  while  not  overdesigning  a  system. 


The  technique  of  warfare  is  always  changing,  sometimes 
drastically;  yet  I’m  afraid  one  must  admit  that  the  military 
mind  is  only  too  often  very  unreceptive  to  new  ideas  and  new 
methods.  The  chivalrous  horror  and  disgust  which  the 
armoured  knights  of  old  felt  for  the  vulgar  invention  of  gun 
powder  was  matched  not  so  many  years  ago  by  similar 
righteous  disgust  at  the  breach  of  gentlemanly  military 
etiquette  by  the  introduction  of  the  aeroplane  the  and  bomb  ... 

I  suspect,  however,  that  the  true  reason  for  the  opposition  to 
the  use  of  gun  powder  was  fear,  fear  that  it  would  upset  the 
established  order  -  as  in  fact  it  did....  It  has  not  been  morality 
but  expediency  that  has  governed  the  use  of  new  weapons. 

-  Arthur  W.  Tedder 


Ignoring  the  gap 

History  is  filled  with  examples  of  great  military  powers  falling  after 
failing  to  adopt  emerging  military  techniques  or  weapons.  Emerging 
military  techniques  and  weapons  have  also  changed  military  culture  and 
some  argue,  military  ethos.23  The  stirrup  and  the  machine  gun  are 
prominent  examples  of  technology  affecting  military  culture  and  ethos. 


23  Michael  R.  Contratto,  “The  Decline  of  Military  Ethos  and  the  Profession  of  Arms:  An  Argument  Against 
Autonomous  Lethal  Engagements,”  (submitted  in  partial  fulfillment  of  requirements  of  the  Air  War 
College,  Air  University,  February  201 1).  One  of  Contratto’s  central  arguments  is  that  by  allowing  machines 
to  kill,  the  US  military  risks  its  military  ethos  and  a  loss  to  the  professionalism  of  arms. 


The  toe  stirrup  allowed  a  rider  to  propel  the  strength  of  a  horse 
through  a  weapon  such  as  a  lance.  Hesitation  across  Europe,  with  the 
exception  of  the  Franks,  to  use  or  to  grasp  the  concept  of  the  toe  stirrup 
was  a  primary  factor  supporting  the  Frank’s  military  dominance.  The  toe 
stirrup,  credited  with  the  establishment  of  feudalism,  also  resulted  in  a 
significant  change  to  military  culture.  24 

The  emergence  of  the  machine  gun  challenged  the  “supremacy  of 
man  as  opposed  to  mere  machine.”  John  Ellis  points  out  that  the 
nineteenth  century  European  officer  corps,  most  from  aristocratic  origins 
and  socially  isolated,  lacked  the  ability  to  conceive  of  war  beyond  its 
existence  in  the  previous  century.  They  rejected  the  use  of  the  machine 
gun  as  outside  the  military  ethos.  The  officers  missed  the  concept  that 
dominance  in  war  was  shifting  towards  “dominance  of  the  tools  of  war. 
For  them  the  war  still  was  an  act  of  will.  Military  memories  and 
traditions  had  been  formed  in  a  pre-industrial  age  when  the  final  bayonet 
or  cavalry  charge  might  be  decisive.  For  them,  in  the  last  analysis,  man 
was  the  master  of  the  battlefield.”25 

The  emergence  of  Unmanned  Aerial  Systems  (UAS),  later  named 
Remotely  Piloted  Aircraft  (RPAs),  in  large  quantities  during  the  early 
twenty-first  century  created  much  consternation  within  the  ranks  of  the 
USAF.  In  a  2005  Air  and  Space  Power  Journal  article,  Hoffman  and 
Kamps  explain  “Some  pilots  appear  wary  of  the  usefulness  of  UAVs  and 
UCAVs,  primarily  because  they  simply  don't  like  the  idea  of  being 
replaced  by  a  robotic  aerial  vehicle.  In  2000  a  military  pilot  told  one  of 
the  authors  that  ‘it  will  be  a  long  time  before  any  of  us  will  be 
comfortable  releasing  bombs  and  betting  the  ranch  using  UAVs.’  Since 
that  time,  we  have  gained  enough  confidence  in  the  UAVs  reliability, 
positioning,  and  target  accuracy  that  slinging  bombs  from  this  aircraft 

24  Lynn  White  Jr.  Medieval  Technology  and  Social  Change  (New  York,  NY:  Oxford  University  Press. 
1962),  28. 

25  John  Ellis,  The  Social  History  of  the  Machine  Gun  (Baltimore,  MD:  The  Johns  Hopkins  University  Press, 
1975),  49-50. 


has  become  a  foregone  conclusion.  However,  the  culture  still  has  a  long 
way  to  go  in  accepting  unmanned  technology  for  other  potential 
missions.”26 

Questioning  new  technology  is  healthy.  Asking  the  wrong 
questions  at  the  peril  of  mis-focused  attention  is  dangerous.  27  The 
assumption  that  an  RPA  was  capable  of  executing  missions  outside  of 
programming  space  hindered  acquisitions  from  design,  development, 
manufacturing,  training,  and  execution.  The  fear  of  change  hid  the  logic 
that  a  fly-by-wire  aircraft  with  a  human  attached  only  allowed  the 
human  a  vote,  perhaps  not  even  a  majority  vote.  Similar  control  systems 
exist  in  RPAs  with  the  exception  of  distance  (interference  in  the  control 
system  due  to  the  distance  was  an  area  overlooked  until  the  video 
downlink  feeds  were  hacked).28  Had  advocates  and  nay-sayers  realized 
the  affect  of  the  mis-focused  discussion,  pertinent  questions  may  have 
concentrated  attention  on  the  video  downlink  vulnerability  sooner.  Key 
intelligence  data  may  not  have  been  compromised.  Hacking  of  Predator 
video  may  have  resulted  from  improperly  focused  attention  driven  by 
ignorance  of  a  technology  gap. 


26  James  C.  Hoffman  and  Charles  Tustin  Kamps,  “At  the  crossroads:  future  ‘manning’  of  unmanned  aerial 
vehicles Air  and  Space  Power  Journal  19,  no.  1  (Spring  2005), 

http://www.freepatentsonline.com/article/Air-Space-Power-Journal/135886989.html  (accessed  April  1 , 
2012).  Additional  source  support  from  Brian  Mockenhaupt  article  “We’ve  seen  the  future  and  it’s 
unmanned,”  Esquire.com ,  http://www.esquire.com/features/unmanned-aircraft-l  109-2  (accessed  April  1, 
2012),  “In  2001,  ninety  years  after  an  airplane  first  dropped  a  bomb,  a  Predator  launched  a  Hellfire  missile 
in  Afghanistan  for  the  first  time.  The  next  year,  a  Predator  fired  a  Stinger  missile  at  an  Iraqi  MiG-25.  The 
missile’s  heat  seeker  was  thrown  off  by  the  MiG’s  missile,  which  destroyed  the  Predator,  but  still:  A  drone 
had  been  in  a  dogfight.” 

27  Thomas  P.  Christie,  “Operational  Test  and  Evaluation  Report  on  the  Predator  Medium- Altitude 
Endurance  Unmanned  Aerial  Vehicle,”  Air  Force  Operational  Test  and  Evaluation  Center, 
http://pogoarchives.org/rn/dp/dp-predator.pdf  (accessed  May9,  2012).  The  report  identifies  the  operational 
short  comings  of  the  Predator  aircraft  when  evaluated  against  user  defined  requirements.  The  Predator  was 
marked  as  “not  operational  effective  or  suitable.”  At  the  time  of  the  Operational  Test  Evaluation  report, 
Predator  aircraft  had  successfully  executed  operational  missions  in  support  of  the  Global  War  on  Terroism. 
Following  this  statement,  the  report  clarifies  “the  disparity  between  the  apparently  successful  fielded 
system  and  the  system  that  did  not  perform  well  in  the  initial  operational  test  and  evaluation  (IOT&E)  is 
largely  attributed  to  the  fact  that  the  fielded  system  is  tasked  and  operated  well  within  known  limitations.” 

28  Sibhan  Gorman,  Yochi  J.  Dreazen,  and  August  Cole,  “Insurgents  Hack  U.S.  Drones:  $26  Software  Is 
Used  to  Breach  Key  Weapons  in  Iraq;  Iranian  Backing  Suspected,”  Wall  Street  Journal  Online,  December 
17,  2009,  http://online.wsj.com/article/SB12610224788909501  l.html  (accessed  May  9,  2012). 


Lt  General  Deptula  brings  the  point  home  regarding  technology 
capability  and  human  trust;  he  “doesn’t  expect  that,  even  with  onboard 
computer  processors  rated  equivalent  to  the  human  brain,  an  air-to-air 
fighter  could  be  remotely  piloted.  So  far,  technology  does  not  allow  the 
‘360-degree  spherical  situational  awareness’  necessary  for  a  pilot  to 
sense  a  rapidly  changing  situation  and  take  the  appropriate  action  in  a 
split-second  battle.  Moreover,  ‘linkages  are  vulnerable,’  and  for  the  near 
term,  RPAs  can’t  be  trusted  yet  to  wield  lethal  power  without  the  over 
watch  of  a  human.”29  While  supposing  possible  a  processor  equivalent  to 
the  human  brain,  General  Deptula  does  not  allow  for  the  consideration  of 
an  autonomous  air-to-air  engagement.  He  defaults  to  the  concept  of 
remotely  piloted  control  of  a  system  with  supposed  capabilities  of  a 
human. 


29  Tirpak,  The  RPA  Boom. 


Chapter  3:  Changing  Societal  Norms 


“The  history  of  technology  is  part  and 
parcel  of  social  history  in  general.  ” 

-John  Ellis 

In  1914,  entrenched  combatants  on  both  sides  of  the  Western 
Front  in  World  War  I  participated  in  a  decision  to  trust  their  enemy 
during  a  period  of  peace  and  celebration  of  Christmas.1  The  WWI 
Christmas  Truce  exemplifies  trust.  It  existed  between  the  combatants 
despite  the  evidence  of  previous  WWI  actions.  Evidence  must  be  set  aside 
for  trust  to  exist.  The  following  chapter  addresses  trust  through  an 
examination  of  social  norms  and  moral  hazard. 

The  judge  began  to  question  if  society  as  a  whole  was  even  aware 
of  how  far  autonomous  systems  penetrated  daily  lives.  He  wondered  why 
his  court  was  used  to  address  the  issue;  why  had  this  issue  not  been 
addressed  as  the  ALEs  developed.  Surely,  the  designers  or  the  users  of 
ALEs  had  asked  these  questions.2  A  thought  then  entered  Judge  Minos’s 
mind.  Had  society  recognized  the  emergence  of  ALEs?  Were  the 
individuals  who  created  the  ALEs  so  tunnel-focused  on  the  task  that  the 
implications  of  their  designs  passed  through  moral  blind  spots?  The 
judge  knew  a  historical  accounting  of  actions  might  not  answer  these 
questions.  His  thoughts  moved  on  to  greater  questions  of  fate, 
determinism,  and  of  individual  choice  within  society.  Was  our  path 


1  “The  Christmas  Truce,”  The  First  World  War.com:  a  Multimedia  history  of  World  War  1. 
http://www.firstworldwar.com/features/christmastmce.htm  (accessed  on  April  23,  2012). 

2  Opinion  papers  addressing  autonomous  liability  have  increased  over  the  years.  In  Liability  for 
Autonomous  Agent  Design,  Carey  Heckman  from  the  Stanford  Law  School  and  Jacob  O.  Wobbrock  from 
the  Stanford  Symbolic  Systems  Program  address  liability  in  three  scenarios  centered  on  autonomous  agents 
employed  throughout  society  in  2012.  Each  scenario  surrounds  actions  taken  by  an  asynchronous 
autonomous  agent  that  “somewhat  controls]  their  own  actions  and  do  not  depend  on  constant  human 
feedback.”  (395)  While  acknowledging  that,  for  now,  “the  causation  for  any  agent  malfunction  lies  with  a 
human  (396),”  the  actions  of  adaptive  agents  become  more  indeterminate  with  evolutionary  growth  (398) 
and  the  result  is  confusion  as  to  who  is  liable  (395).  Additionally  the  automotive  industry  is  struggling  with 
liability  concerning  autonomous  vehicles.  In  Legal  Aspects  of  Autonomous  Driving 


predetermined  by  the  growth  of  technology  or  did  societal  intent  and 
individual  responsibility  construct  the  situation? 

Social  Norms 

Social  constructionist  views  and  technological  determinism  are  two 
extremes  that  bound  the  following  discussion  on  social  norms.3  As  in 
most  social  cases,  neither  extreme  is  complete  without  consideration  of 
the  other.  The  views  that  society  is  deterministically  guided  or  that 
thoughtful  intent  and  action  construct  society  fall  in  the  realm  of 
philosophy  as  much  as  historical  reference.  The  relevance  of  this 
philosophical  debate  concerns  the  emergence  of  autonomous  systems. 

A  technological  deterministic  view  tends  to  the  emergence  of  ALEs 
without  societal  forethought.  A  socially  constructed  viewpoint  would 
suggest  that  ALEs  must  be  a  recognized  cognitive  decision  for  their 
emergence  to  occur.  Technological  momentum,  a  middle  ground, 
supports  and  tempers  both  views.  Momentumist  thought  does  not 
require  the  strict  outcome  of  determinism  but  it  does  support  the  likely 
progression  of  technology.  4 

Beyond  the  philosophical  drivers,  what  changes  will  occur  for  the 
integration  of  autonomous  systems  into  society?  Must  recognition  of  the 
autonomous  nature  occur  before  governance  changes  or  is  it  more  likely 
that  ALEs  will  emerge  and  require  governance  changes  without  general 
recognition  from  society? 

The  integration  of  autonomous  systems  into  the  daily  lives  of 
society  will  likely  have  secondary  effects.  Is  there  a  way  to  predict  and 
leverage  these  effects  as  autonomous  systems  develop?  Can  we  use 


3 

Wiebe  E.  Bijker,  Thomas  P.  Hughes,  and  Trevor  Pinch,  eds.  ,The  Social  Construction  of  Technological 
Systems ,  (Cambridge  MA:  The  MIT  Press,  1989). 

4  Thomas  P.  Hughes,  “Technological  Momentum,”  in  Does  Technology  Drive  History?  The  Dilemma  of 
Technological  Determinism ,  eds.  Merritt  Roe  Smith  and  Leo  Marx  (Cambridge  MA:  The  MIT  Press, 
1994),  101-113. 


secondary  effects  as  a  tipping  point  to  recognize  autonomous  systems 
from  traditional  programmed  and  confined  systems? 

Individual  perspective  of  situational  context  drives  the  morality  of 
decisions.5  In  general,  killing  is  viewed  as  immoral.  However,  killing  in 
self-defense  or  to  protect  a  weaker  person  may  be  viewed  as  morally 
correct.  Considering  the  range  of  individual  interpretation  of  context, 
does  a  rational  view  of  morality,  removed  from  temporal  emotion,  provide 
room  for  an  expanded  discussion  of  the  morality  of  autonomous  system 
decisions  over  human? 

Modern  examples  of  anti-lock  braking  systems  as  well  as  the  use  of 
precision  guided  munitions  provide  examples  of  technology  exceeding 
human  capability  and  rationally  challenge  the  moral  decision  to  not  use 
technologically  aided  systems.  Kant  argues  for  an  evaluation  of  morality 
viewed  through  pure  reason;  “as  and  so  with  all  the  other  moral  laws 
properly  so  called;  that,  therefore,  the  basis  of  obligation  must  not  be 
sought  in  the  nature  of  man,  or  in  the  circumstances  in  the  world  in 
which  he  is  placed,  but  a  priori  simply  in  the  conception  of  pure 
reason.”6  Are  autonomous  agents,  free  from  human  emotion,  capable  of  a 
more  pure  reason  and  thus  considered  more  moral? 

One  can  imagine  the  following  response  in  an  amicus  curiae  brief: 


5  Sean  Cort,  “Content  versus  Context:  It’s  not  what  you  say,  it’s  how  you  say  it.’’  The  Power  of  Perspective 
(December  7,  2009),  http://www.psychologytoday.com/blog/the-power-perspective/200912/content-vs- 
context  (accessed  on  April  24,  2012).  Cort’s  article  describes  the  effect  of  both  content  and  context  on 
perspective.  Since  morality  is  interpreted  through  individual  perspective,  morality  must  also  be  affected  by 
content  and  context. 

6  Kant,  Fundamental  Principals  of  the  Metaphysic  of  Morals,  http://www.  manybooks.  net/  support/  k/ 
kantimma /  kantimmaetext04ikfpml0iliad.pdf  (accessed  on  February  22,  2012), 7. 


Honorable  Minos, 

Your  questions  regarding  the  constructed  or 
determined  nature  of  socially  integrated  autonomous 
systems  are  well  founded.  If  I  understood  you  correctly,  your 
intention  exceeds  a  mere  academic  or  philosophical  debate 
concerning  the  extremes  of  social  construction  and 
technological  determination.  You  seek  to  understand  the 
ownership  of  intention.  If  society  intended  the  development 
of  ALEs  we  must  hold  society  accountable  to  and  for  ALEs.  If 
determinism  drove  ALE  existence,  must  society  be  held 
accountable  to  the  ALEs  or  will  the  ALE  deterministic  nature 
care  for  the  evolution  of  ALEs  without  taxing  society? 

In  either  case,  whether  ALEs  were  intended  or 
deterministically  evolved,  the  ALEs  must  be  recognized  and 
governance  must  change  to  incorporate  ALEs  into  society.  A 
historical  review  of  the  relationship  between  recognition  and 
governance  will  support  your  questions  concerning 
constructed  or  determined  nature  of  the  technology.  Looking 
directly  at  a  target  often  blinds  our  vision.  We  can  be 
overwhelmed  by  content  and  lose  sight  of  the  context. 
Secondary  effects  may  provide  you  with  a  better  vision  of  the 
socially  accepted  autonomy  of  machines. 

As  you  build  your  opinion  for  the  legal  status  of 
autonomous  systems,  keep  in  mind  the  philosophical 
question  of  could  autonomous  systems,  as  true  rational 
actors,  be  more  ethical  than  humans?7  Providing  legal 
standing  to  a  machine  will  add  weight  to  this  philosophical 
debate  but  it  will  also  force  a  discourse  on  practical 
questions  concerning  trust,  ethics,  and  reliability.  Is  it 

7 

Kant,  Fundamental  Principals  of  the  Metaphysic  of  Morals.  Kant  provides  and  argument  for  a  pure 
theoretical  rationality  that,  while  envisioned  by  humans,  exceeds  human  capability. 


unethical  to  not  use  an  autonomous  machine  if  the  machine 
has  demonstrated  greater  reliability  to  save  lives  than  the 
human  counterpart?  By  assigning  legal  standing,  you  will 
force  questions  of  trust  and  the  recognition  of  legitimacy 
related  to  reliability  and  ethics. 

With  regards, 

R.  Kantakins8 

Socially  constructed  or  technologically  determined? 

Much  debate  exits  concerning  the  sociological  formation  of  our 
society  and  the  integration  of  technology.  Are  we  destined  to 
continuously  push  technological  barriers  without  regard  to  moral 
implications  or  is  our  hubris  great  enough  to  believe  that  individual 
intention  fuels  societal  construction? 

Post  modernism  strengthens  the  view  of  social  constructionist’s 
reign  over  technological  progression.  Four  case  studies  provided  by 
Hughes  indicate  that  social  construction  bounds  large  US  technological 
programs.9  As  the  postmodern  world  evolved  in  the  United  States,  the 
populous  challenged  government-led  programs  to  conform  to  moral  and 
safety  standards  not  compelled  prior  to  World  War  II.  Such  a  post¬ 
modernist  view  of  society  logically  indicates  that  the  desire  for  social 
safety  and  other  advantages  gained  from  autonomous  systems  will  drive 
society  to  intentionally  construct  circumstances  encouraging  the 
development  of  autonomous  systems.  One  modern  example  of 
constructed  circumstances  surrounds  the  perceived  safety  gains  of 


8  The  thesis  author  used  artistic  license  to  represent  a  conglomerate  of  opinions  from  Kant  and  Akins.  The 
letter  is  signed  with  a  fictitious  name. 

9  Thomas  P.  Hughes,  Rescuing  Prometheus:  Four  Monumental  Projects  that  Changed  the  Modern  World 
(New  York:  Vintage  Books,  2000),  303-305. 


autonomous  collision  avoidance  in  cars.10  However,  as  most 
environmentalists  will  attest,  socially  minded  concerns  do  not  always 
translate  to  socially  conscious  development. 

Pinch  and  Bijker’s  view  of  social  constructivism  relies  on  the  social 
groups  that  are  “concerned  with  the  artifact  and  the  meanings  that  those 
groups  give  to  the  artifact.  A  problem  is  defined  as  such  only  when  there 
is  a  social  group  for  which  it  constitutes  a  ‘problem.’”* 11  In  this  light,  a 
social  constructionist  could  find  a  world  integrated  with  autonomous 
systems  without  specific  societal  intention.  A  group  must  be  aware  of 
and  identify  a  technological  artifact  as  “constituting  a  problem”  for  a 
scheme  to  develop.12  Once  a  social  group  has  identified  a  problem,  it  is 
then  possible  to  address  several  conflicts  ranging  from  “technical 
requirements  by  different  social  groups  . . .  conflicting  solutions  to  the 
same  problem  ...  and  moral  conflicts.  Within  this  scheme  various 
solutions  to  these  conflicts  and  problems  are  possible  -  not  only 
technological  ones  but  also  judicial  or  even  moral  ones.”13  Regardless  of 
how  autonomous  systems  progress,  in  the  eyes  of  a  social 
constructionist,  the  ALE  trial  represents  a  recognized  problem  for  a 
social  group  to  address.  The  trial  becomes  the  condition  required  for  the 
next  phase  of  ALE  recognition. 

Pinch  and  Bijker’s  artifact  recognition  requirement  is  similar  to  the 
ideas  of  technological  determinism.  Technological  determinists  view  the 
“efficacy  of  technology  as  a  driving  force  of  history:  a  technical  innovation 


10Tom  Vanderbilt,  “Let  the  Robot  Drive:  The  Autonomous  Car  of  the  Future  Is  Here,”  Wired  (February 
2012  edition,  posted  January  20,2012),  http://  www.  wired,  com/  magazine/  2012/  01/  ff_  autonomouscars/ 
all/  l(accessed  January  2012). 

11  Trevor  J.  Pinch  and  Wiebe  E.  Bijker,  “The  Social  Construction  of  Facts  and  Artifacts:  Or  How  the 
Sociology  of  Science  and  the  Sociology  of  Technology  Might  Benefit  Each  Other,”  in  The  Social 
Construction  of  Technological  Systems,  ed.  by  Wiebe  E.  Bijker,  Thomas  P.  Hughes,  and  Trevor  Pinch, 
(Cambridge  MA:  The  MIT  Press,  1989),  30.  Referenced  hereafter  as  Pinch  and  Bijker  “The  Social 
Construction  of  Facts  and  Artifacts.” 

12  Pinch  and  Bijker,  “The  Social  Construction  of  Facts  and  Artifacts,”  30. 

13  Pinch  and  Bijker,  “The  Social  Construction  of  Facts  and  Artifacts,”  35-39. 


suddenly  appears  and  causes  important  things  to  happen.”14  Robert 
Heilbroner  captured  this  sentiment  in  his  essay,  “Do  Machines  Make 
History?”  Heilbroner  represents  a  soft  determinist  approach.  His  main 
thesis  concerns  “the  effect  of  technology  in  determining  the  nature  of  the 
socioeconomic  order,”  and  highlights  the  focus  of  determinism  of 
technology  as  the  driving  force. 15  Heilbroner  uses  the  concept  of 
simultaneous  discovery  or  technological  clustering  (“the  process  of 
discovery  takes  place  along  a  well-defined  frontier  of  knowledge  rather 
than  in  grab-bag  fashion”)  to  support  a  claim  that  “technological 
evolution  follows  a  sequential  and  determinate  rather  than  random 
course.”16  If  societal  construction  drove  socioeconomic  conditions  rather 
than  technology,  the  variation  of  global  societies  would  not  account  for 
clusters  of  simultaneous  technological  advancement  over  great  distances. 

The  intermittent  American  technocracy  movement  highlights  an 
American  tendency  towards  technological  determinism. 17  The  industrial 
revolution,  fueled  by  necessities  of  efficient  labor  through  machinery, 
“glorified  the  march  of  invention  and  the  material  progress  of  the  age. 
Technology  and  science  not  only  became  the  great  panacea  for  everyday 
problems;  they  also  stood  for  values  at  the  core  of  American  life.”18 

A  distinction  can  be  made  to  separate  the  determinist  development 
of  a  technology  from  the  application  of  the  technology.  Lynn  White,  Jr. 
explains  that  “a  new  device  merely  opens  a  door;  it  does  not  compel  one 
to  enter.  The  acceptance  or  rejection  of  an  invention,  or  the  extent  to 
which  its  implications  are  realized  if  it  is  accepted,  depends  quite  as 


14  Leo  Marx  and  Merritt  Roe  Smith,  “Introduction”  in  Does  Technology  Drive  History?  The  Dilemma  of 
Technological  Determinism,  ed.  Merritt  Roe  Smith  and  Leo  Marx  (Cambridge  MA:  The  MIT  Press,  1994), 
xi. 

15  Robert  L.  Heilbroner,  “Do  Machines  Make  History?”  in  Does  Technology  Drive  History?  The  Dilemma 
of  Technological  Determinism,  ed.  Merritt  Roe  Smith  and  Leo  Marx,  53. 

16  Heilbroner,  “Do  Machines  Make  History?”  56. 

17  John  Ellis,  The  Social  History  of  the  Machine  Gun  (Baltimore,  MD:  The  John  Hopkins  University  Press, 
1975),  22-23. 

18  Marx  and  Smith,  Does  Technology  Drive  History?  The  Dilemma  of  Technological  Determinism ,  23. 


much  upon  the  condition  of  a  society,  and  upon  the  imagination  of  its 
leaders,  as  upon  the  nature  of  the  technological  item  itself.”19  White 
illustrated  the  point  with  the  stirrup.  “The  Anglo-Saxons  used  the 
stirrup,  but  did  not  comprehend  it;  and  for  this  they  paid  a  fearful  price. 
It  was  the  Franks  alone  -  presumably  led  by  Charles  Martel’s  genius  - 
who  fully  grasped  the  possibilities  inherent  in  the  stirrup  and  created  in 
terms  of  it  a  new  type  of  warfare  supported  by  a  novel  structure  of 
society  which  we  call  feudalism.”20  Emergence  of  technology  requires  a 
user  to  grasp  the  possibilities  inherent  in  the  technology.  A  determinist 
may  support  the  notion  that  thermonuclear  war  became  inevitable  once 
physicist  conceived  of  the  idea  to  harness  the  energy  of  a  split  atom,  but 
human  choice  not  to  use  thermo-nuclear  weapons  is  an  example  of  a 
determined  technology  intentionally  not  used. 

Technological  momentum  tends  towards  technological 
deterministic  behavior  while  considering  socially  constructed  control. 

This  middle  ground  acknowledges  the  intention  driving  socially 
constructed  systems.  Nonetheless,  a  greater  impulse  of  intention  is 
required  to  change  the  momentum  of  what  seems  to  be  a  determined 
path. 

Hughes  summaries  the  three  approaches  by  defining  technological 
determinism  as  “the  belief  that  technical  forces  determine  social  and 
cultural  changes.  Social  construction  presumes  that  social  and  cultural 
forces  determine  technical  change.  Technological  momentum  infers  that 
social  development  shapes  and  is  shaped  by  technology.  Momentum  is 
time  dependent.”21  The  trial,  dubbed  Machine  on  Trial,  can  slow  the 
momentum  by  manipulating  the  time  required  for  further  ALE 
development.  The  trial  also  represents  a  possible  pivot  point  for  the  social 


19  Lynn  White  Jr.  Medieval  Technology  and  Social  Change  (New  York,  NY:  Oxford  University  Press, 
1962),  28. 

20  White,  Medieval  Technology  and  Social  Change,  28. 

21  Thomas  P.  Hughes,  “Technological  Momentum,”  102. 


constructionists  to  correct  an  already  emerged  technology  fueled  into 
existence  through  technological  momentum. 


If  the  radiance  of  a  thousand  suns 
Were  to  burst  at  once  into  the  sky, 

That  would  be  like  the  splendor  of  the  Mighty  One... 

I  am  become  Death, 

The  shatterer  of  Worlds. 

-  The  Bhagavad-Gita 


Change  required  for  ALBs  to  operate 

Recognition  and  governance  drive  the  changes  required  within 
society  for  ALEs  to  operate.  Must  society  recognize  the  ALE  as  an 
autonomous  system  before  it  is  possible  for  the  changes  to  occur,  or  is  it 
more  likely  that  ALEs  will  emerge  without  general  recognition  from 
society  -  along  the  lines  of  a  slowly  boiling  frog?22 

Kurzweil,  through  several  essays,  addresses  the  human  tendency 
to  dismiss  significant  leaps  in  artificial  intelligence.  He  states  “a  common 
reaction  to  the  proposition  that  computers  will  seriously  compete  with 
human  intelligence  is  to  dismiss  this  specter  based  primarily  on  an 
examination  of  contemporary  capability.”23  Kurzweil  describes  his 
concept  of  an  intuitive  linear  view  as  the  human  tendency  to  assume  the 
current  rate  perceived  progress  to  remain  constant.  The  recognition  of  an 
exponential  technological  growth  curve  reinforces  the  intuitive  linear  view 
when  the  time  period  is  viewed  briefly.24  Debates  surrounding  the 
timescale  required  for  self- replicating  entities  ranges  from  never,  to  100 
years,  to  25  years.  Kurzweil  defends  both  positions  of  100  and  25  years 


22  The  Boiled  Frog,  http://allaboutfrogs.org/stories/boiled.html  (accessed  May  9,  2012).  “This  parable  is 
often  used  to  illustrate  how  humans  have  to  be  careful  to  watch  slowly  changing  trends  in  the  environment, 
not  just  the  sudden  changes.” 

23  Ray  Kurzweil,  The  Age  of  Spiritual  Machines  (New  York,  NY :  Penguin  Books,  2000),  4. 

24  Ray  Kurzweil,  “The  Law  of  Accelerating  Returns,”  in  Accelerating  Intelligence  Essays  (March  7,  2001), 
http://www.kurzweilai.net/the-law-of-accelerating-retums  (accessed  on  March  20,  2012).  The  slope  of  an 
exponential  curve  appears  linear  when  the  change  in  one  axis  over  the  other  is  considered  over  a  short 
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but  cautions  that  the  discrepancy  stems  from  a  misperception  of  the 
acceleration  of  the  rate  of  evolutionary  and  technological  progress.  He 
states  that  100  years  of  today’s  progression  rate  (stated  in  2001)  is 
equivalent  to  25  actual  years.25  If  Kurzweil’s  intuitive  linear  thinking 
plagues  the  human  race,  humanity  may  feel  a  scalding  sensation  without 
recognizing  that  the  burn  stems  from  the  boiling  water  surrounding 
society  as  the  ever-increasing  rate  of  autonomous  systems  integrates  into 
daily  life. 

The  atomic  era  serves  as  a  warning  for  technology  advancement 
with  respect  to  acceptable  governing  behavior.  Six  years  to  the  day 
passed  from  Einstein’s  decision  to  write  President  Roosevelt  advocating 
support  of  the  Manhattan  Project  until  the  first  atomic  weapon  ignited 
over  the  Trinity  Test  Site  on  July  16,  1945. 26  The  technological 
advancement  leading  first  to  the  atomic  weapons  followed  quickly  by 
thermonuclear  weapons  is  one  example  of  accelerated  technology 
outpacing  politics  and  governance.  President  Truman  understood  the 
military  capacity  of  atomic  weapons  when  he  ordered  the  release  of  two 
over  Japanese  cities.  However  it  took  years  for  American  political  leaders 
to  understand  the  political  implications  of  the  weapons.27  Technology 
advanced  at  a  far  greater  rate  than  domestic  or  international  governance. 
The  destructive  power  of  nuclear  weapons  required  political  powers  to 
proceed  cautiously.  How  much  damage  might  have  occurred  if  the 
destructive  power  of  nuclear  weapons  was  not  internalized  in  time  to 
slow  the  exploitation  of  the  weapons  in  the  years  when  governance  was 
solidifying?  Consider  autonomous  system  development.  The  benign 
nature  of  these  systems  will  not  likely  spike  the  concern  of  nuclear 
weapons  while  the  time  required  to  advance  autonomous  systems  is 


25  Kurzweil,  The  Law  of  Accelerating  Returns ,  1-2,  43-46. 

26  The  History  Channel,  10  Days  that  Unexpectedly  Changed  America, 
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27  Campbell  Craig,  Destroying  the  Village:  Eisenhower  and  Thermonuclear  War  (New  York,  NY: 
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projected  to  dwarf  the  time  required  to  normalize  nuclear  development. 

In  1 5  Minutes  to  Nuclear  Annihilation,  Keeney  highlights  the  evolution  of 
nuclear  control  from  a  chain  of  complete  civilian  authority  and  physical 
control  to  a  military-controlled  chain  required  to  meet  Strategic  Air 
Command’s  15  minute  alert  status.28 

A  technological  determinist  would  view  the  progression  of  ALE 
development  as  one  of  many  evolutionary  steps.  From  a  determinist  view, 
the  laws  must  change  to  reflect  progression  of  technology.  Incompatible 
laws  will  not  stifle  the  determinate  nature  of  technology  development  for 
very  long.  A  social  constructionist  may  view  the  creation  and  refinement 
of  laws  and  governance  as  a  precursor  to  the  development  of  ALEs. 
Governance  is  one  means  used  to  construct  technological  advances  into 
society.  Technological  momentumists  may  see  the  development  of  ALEs 
in  close  step  with  the  expansion  of  governance  to  head  off  technological, 
and  therefore  societal,  stifling.  All  three  views  require  active  integration, 
leading  or  following,  of  autonomous  systems  into  governance. 

It  does  not  seem  to  matter  if  social  constructivism,  technological 
momentum,  or  technological  determination  drives  the  recognition  and 
governance  of  ALEs.  All  three  philosophical  views  allow  for  the  existence 
of  the  current  trial.  The  effect  of  the  path  society  takes  may  not  drive  the 
existence  of  the  trial  but  it  may  drive  the  questions  raised  during  the  trial. 
Are  the  ALEs  creations  of  destiny  or  of  free  will?  Does  the  manner  of 
creation  lend  towards  recognition  of  rights  or  sentient  life?  Are  the  ALEs 
moral  agents  holding  to  both  responsibility  and  accountability,  or  are  the 
ALEs  simply  an  extension  of  a  corporation  -  an  entity  that  takes  action 
and  thus  is  accorded  rights  without  thought  or  intent? 


28  L.  Douglas  Keeney,  15  Minutes  to  Nuclear  Annihilation:  General  Curtis  LeMay  and  the  Countdown  to 
Nuclear  Annihilation  (NY:  St  Martin’s  Press,  201 1). 


Recognition  of  possible  secondary  effects? 

The  Internet  was  “created  for  research  purposes,  for  the  exchange 
of  ideas,  not  for  commerce,  where  money  would  change  hands,  or  for  the 
purposes  of  controlling  critical  systems.”29  The  initial  group  that  set  up 
the  ARPANET  did  not  foresee  secondary  effects  of  the  networked  system 
and  thus  did  not  install  within  it  contingencies  for  security  or  expansion 
beyond  four  billion  Internet  protocol  addresses.  It  is  possible  for 
secondary  effects  to  exceed  the  impact  of  primary  intentions. 

Kurzweil  paints  a  picture  of  the  human  race  and  machines 
evolving  into  one  entity.  He  proposes  that  autonomous  artificial 
intelligence  will  lead  to  secondary  effects  and  the  evolutionary  extension 
of  human  life.30  Are  there  secondary  effects  society  might  recognize  that 
are  not  as  pronounced  as  the  thousand-year  human  life  extension 
discussed  by  Kurzweil  yet  are  still  profound  enough  to  alter  human 
society  such  as  the  effects  driven  by  the  Internet?  In  short,  can  society 
identify  secondary  effects  that  may  trigger  recognition  of  the  autonomous 
age? 

Autonomous  systems  of  varying  levels  of  independence  and 
intelligence  are  prevalent  throughout  modern  industrialized  societies. 
Autonomous  agents  execute  intelligent  computer  programs  designed  to 
increase  efficiency  of  network  systems.  Google’s  intelligent  and 
personalized  search  engine  in  an  example  of  tailored  learning  algorithms 
proliferated  across  a  mass  market.  The  2007  announcement  to  default 
Google  user  accounts  to  a  personalized  mode  was  “perhaps  the  most 
significant  change  in  search  marketing  in  the  past  few  years.”31 

Debates  concerning  privacy,  search  engine  anarchy,  or  unfair 
business  practices  did  not  diminish  Google’s  world  predominance  as  a 
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search  engine.32  With  the  explosive  growth  of  the  Internet  the  need  for 
ever  more  efficient  search  and  organization  algorithms  increases.  Google 
may  be  the  most  prolific  search  algorithm  but  it  is  not  unique.  A  2002 
Conference  on  Information  and  Knowledge  Management  (CIKM)  paper 
evaluated  methods  to  personalize  internet  searches  with  the  following 
technique:  “A  user  profile  and  a  general  profile  are  learned  from  the 
user's  search  history  and  a  category  hierarchy  respectively.  These  two 
profiles  are  combined  to  map  a  user  query  into  a  set  of  categories. 

Several  learning  and  combining  algorithms  are  evaluated  and  found  to  be 
effective.  Among  the  algorithms  to  learn  a  user  profile,  we  choose  the 
Rocchio-based  method  for  its  simplicity,  efficiency  and  its  ability  to  be 
adaptive.”33 

Autonomous  systems  regulate  power  grids,  water  stations,  and 
monitor  and  maintain  household  appliances.  A  growing  list  of  companies 
has  invested  in  “smart  grid”  compatibility  and  technologies  hedging  on 
the  day  when  cities  and  individual  households  will  rely  on  autonomous 
agents  to  control  major  utilities.34  Autonomous  systems  enter  society  to 
improve  convenience  or  efficiency.  Society  becomes  reliant  on  the 
technology  and  eventually  dependent  on  the  autonomous  systems. 

One  definition  of  disruptive  technology  is  a  technology  that 
replaces  and  causes  the  elimination  of  the  previous  technology.  Global 
Positioning  Systems  (GPS),  for  example,  have  spawned  several  such 
secondary  effects.  These  include  advances  of  real  time  logistics  to 
inclusion  of  satellite  navigation  software  as  standard  equipment  in  some 
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automobiles.  Another  example  of  a  secondary  effect  driven  by  GPS 
proliferation  is  drawn  from  the  widespread  use  of  cell  phones  in  the 
United  States.  Prior  to  cell  phones,  functional  pay  phones  existed 
throughout  most  American  cities.  In  2011,  fewer  than  five-hundred 
thousand  payphones  were  operational  in  the  United  States  compared  to 
over  two  million  in  the  1990s.35  Cell  phones  entered  society  as  a  novelty, 
became  a  convenience,  and  within  a  few  years  social  reliance  on  cell 
phones  resulted  in  the  removal  of  most  pay  phones,  and  today  are 
significantly  lessening  occurrences  of  land-wired  home  and  business 
telephone  connections.  Of  note,  as  cell  phones  are  reliant  on  GPS  to 
provide  accurate  timing  signals,  the  widespread  use  of  cell  phones  is 
partially  based  on  the  availability  of  GPS.  In  a  direct  line  of  cause  and 
effect,  one  secondary  consequence  of  the  GPS  constellation  is  the 
removal  of  pay  phones. 

I  see  a  manless  Air  Force  . . .  For  twenty  years  the  Air  Force 
was  built  around  pilots,  pilots,  and  more  pilots  ...  The  next  Air 
Force  is  going  to  be  built  around  scientists  -  around 
mechanically  minded  fellows. 

-  General  “Hap”  Arnold 

The  moral  use  of  humans  when  technology  is  safer 

Should  the  morality  of  a  decision  hold  up  when  viewed  out  of 
context?  Humans  base  decisions  on  a  combination  of  emotional  and 
rational  factors.  Empathy  in  society  allows  for  a  general  understanding  of 
morality.  The  fine  edges  of  moral  right  and  wrong  are  the  battleground 
for  what  is  fair  and  just.  Kant  argues  that  only  decisions  based  on  pure 
reason,  removed  from  emotions  driven  by  the  situation,  allows  for  true 
morality. 


35  Mike  Irwin,  “Pay  phone  graveyard  in  basement  of  Wenatchee  firm,”  The  Wenatchee  World  (October  17 
201 1),  http://www.  komonews.  com/  news/  local/  131990003.html  (accessed  on  March  21,  2012). 


Since  the  1970s  automobile  manufacturers  have  steadily  released 
autonomous  features  and  programs  leading  towards  a  fully  autonomous 
vehicle.  Safety,  efficiency,  and  mobility  drive  the  automation  movement. 
In  2009,  automobile  crashes  accounted  for  the  deaths  of  1.2  million 
people  worldwide  and  34,000  Americans.  At  95%,  driver  error  far  exceeds 
road  conditions  and  technical  failure  as  the  primary  cause  of  these 
accidents.36 

The  introduction  of  automated  systems,  such  as  anti-lock  breaking 
systems  (ABS),  assisted  humans  to  reduce  fatal  crashes.  In  2003,  the  US 
Department  of  Transportation  and  Safety  Administration  in  cooperation 
with  Transport  Canada  conducted  a  motorcycle  ABS  research  project. 

The  ABS  runs  resulted  in  consistent  data  indicating  “an  overall  reduction 
in  stopping  distance  of  5%”  compared  with  non-ABS  assisted  breaking.37 
The  report  noted  that  riders  “did  not  require  significant  experience  or 
special  skill  in  order  to  achieve  a  high  level  of  performance”  with  ABS, 
however  “test  results  from  non-ABS  motorcycles  were  noticeably  more 
sensitive  to  rider  performance  variability.”38  In  addition  to  safety 
concerns,  autonomous  systems  promise  increases  to  efficiency  through 
reductions  in  road  congestion  and  decreased  fuel  consumption,  as  well 
as  offering  greater  flexibility  for  mobility  choices  to  the  elderly  or  youth.39 
At  what  point  does  a  vehicle  driver  become  morally  responsible  to  ensure 
a  reasonable  level  of  safety  prudence  when  opting  to  purchase  a  vehicle 
with  or  without  ABS? 


36  Sven  A  Beiker  and  M  Ryan  Calo,  “Legal  Aspects  of  Autonomous  Driving:  The  need  for  a  legal 
infrastructure  that  permits  autonomous  driving  in  public  to  maximize  safety  and  consumer  benefit,”  Center 
for  Automotive  Research  at  Stanford  (CARS)  and  the  Stanford  University  Center  or  Internet  and  Society 
(CIS),  Stanford  University  Press  (October  2010),  3. 

37  Donovan  Green,  A  Comparison  of  Stopping  Distance  Performance  for  Motorcycles  Equipped  with  ABS, 
CBS,  and  Conventional  Hydraulic  Break  Systems,  United  States  Department  of  Transportation,  National 
Highway  Traffic  Safety  Administration,  pdf  as  presented  at  the  International  Motorcycle  Safety  Conference 
in  Long  Beach,  CA  March  28  -  March  30,  2006,  1-3. 

38  Donovan  Green,  A  Comparison  of  Stopping  Distance  Performance  for  Motorcycles  Equipped  with  ABS, 
CBS,  and  Conventional  Hydraulic  Break  Systems,  3. 

39  Beiker  and  Calo,  Legal  Aspects  of  Autonomous  Driving,  4-5. 


Is  a  vehicle  driver  morally  responsible  in  the  given  scenario?  A 
driver  purchases  an  automobile  on  which  ABS  is  a  standard  feature.  The 
driver  elects  to  remove  the  ABS  feature  from  the  vehicle  due  to  a  personal 
distrust  of  the  ABS  compounded  with  beliefs  that  non-automated 
breaking  assistance  is  a  purer  form  of  driving.  Months  later,  the  driver, 
while  operating  well  within  the  legal  limits  of  the  road,  strikes  a  child 
who  has  crossed  into  the  street.  The  reaction  time  of  the  driver  results  in 
the  death  of  the  child.  If  the  vehicle  had  been  equipped  with  ABS  and  the 
driver  exhibited  the  same  reaction  time,  the  vehicle  would  have  stopped 
several  feet  from  the  child.  Is  the  driver’s  moral  responsibility  for  the 
accidental  death  of  the  child  affected  by  the  decision  to  not  use 
standardized  ABS?  At  what  point  does  it  become  morally  unacceptable  to 
choose  a  human  over  a  technological  solution  once  the  technical  solution 
has  been  proven  significantly  and  consistently  more  capable? 

In  war,  combatant  parties  “must  choose  the  means  and  methods  of 
attack  with  the  aim  of  minimizing  incidental  civilian  losses  and 
damage.”40  This  element  of  just  war  execution  is  explicitly  called  out  in 
Article  57  (1)  of  Additional  Protocol  I,  which  states,  “In  the  conduct  of 
military  operations,  constant  care  shall  be  taken  to  spare  the  civilian 
population,  civilians  and  civilian  objects.”41  Precision  guided  munitions 
accounted  for  100%  of  the  munitions  used  by  NATO  in  the  execution  of 
Operational  Unified  Protector,  the  March  -  October  2011  Libyan  civilian 
protection  effort.42  The  success  of  precision  guided  munitions  in  Desert 
Storm  and  Kosovo  set  the  American  standard  for  acceptable  munitions 


40  Laurie  R.  Blank,  “Targeted  Strikes:  The  Consequences  of  Blurring  the  Armed  Conflict  and  Self-Defense 
Justification,”  37  Win.  Mitchell  L.  Rev.  (201 1),  18. 

41  International  Humanitarian  Law  -  Treaties  &  Documents,  “Protocol  Additional  to  the  Geneva 
Conventions  of  12  August  1949,  and  relating  to  the  Protection  of  Victims  of  International  Armed  Conflicts 
(Protocol  I),  8  June  1977,”  http://www.icrc.org/ihl.nsfAVebART/470-75007370penDocument  (accessed 
May  9,  2012). 

42  Lt  Gen  Ralph  J  Jodice  II,  OUP  CFACC,  Op  Unified  Protector  (OUP)  Mission  Brief,  as  presented  to  the 
School  of  Advanced  Air  and  Space  Studies  (February  2012),  43. 


use  around  civilian  populations.43  Given  the  increase  from  22%  in  the 
Desert  Storm  Gulf  War  to  64%  during  Kosovo  operations  ending  with 
100%  of  precision  guided  munitions  in  the  2010  -  ongoing  Libyan 
operations,  it  is  unlikely  that,  in  wars  of  choice,  US  will  drop  non¬ 
precision  guided  munitions  near  civilian  population  centers  again. 
Precision  munitions  set  a  new  moral  standard  for  acceptable  care  to 
spare  civilian  populations.  The  level  of  automation  in  precision-guided 
munitions  is  significantly  less  than  the  automation  of  an  ALE,  however, 
these  munitions  provide  a  stepping  stone  indicating  how  technologies  are 
capable  of  driving  social  and  moral  acceptance  values. 

The  above  examples  presuppose  a  rational  vision  of  morality.  Kant 
pushed  the  extremes  of  rationality  through  an  “investigation  and 
establishment  of  the  supreme  principle  of  morality.”44  He  argues  for  a 
theoretical  split  of  reason  from  science,  separating  the  empirical  from  the 
rational  nature  of  science.45  By  isolating  the  purely  rational  aspect  of  the 
nature  of  science,  Kant  made  the  theoretical  argument  that  laws  of  moral 
force  are  not  contextually  based.  They  are  “not  valid  for  men  alone  as  if 
other  rational  beings  had  no  need  to  observe  it  . . .  The  basis  of  obligation 
must  not  be  sought  in  the  nature  of  man,  or  in  the  circumstances  in  the 
world  in  which  he  is  placed,  but  a  priori  simply  in  the  conception  of  pure 
reason.”46  If  context  does  not  drive  morality,  how  can  humans,  whose 
emotional  existence  is  based  in  context,  know  pure  morality? 


43  Gulf  W ar  Air  Power  Survey  Staff,  dr.  Eliot  A  Cohen  Director,  Gulf  War  A  ir  Power  Survey  Volume  V:  A 
statistical  compendium  and  chronology ,  Washington  DC  1993,  pages  418,  513-514.  And  Benjamin  S 
Lambeth  “Chapter  5:  Accomplishments  of  the  War”  in  NATOs  Air  War  for  Kosovo:  A  Strategic  and 
Operational  Assessment,  RAND  2001,  87-88.  Of  the  41,309  “strikes”  during  the  Gulf  War,  9,1 17  were  hit 
with  precision  guided  munitions  for  a  total  of  22%.  The  panel  described  a  “strike”  as  a  target  hit  by  one  or 
more  sets  of  munitions.  This  comparison  assumes  strikes  and  aim  points  are  relatively  comparable.  Of  the 
9,815  aim  points  in  NATOs  Air  War  for  Kosovo,  64%  were  hit  with  precision  guided  munitions. 

44  Immanuel  Kant,  Fundamental  Principals  of  the  Metaphysic  of  Morals,  1785,  as  reprinted  in  Matthew 
Stapleton,  The  Project  Gutenberg  EBook  of  Fundamental  Principals  of  the  Metaphysic  of  Morals,  May, 
2004  [EBook  #5682  translated  by  Thomas  Kingsmill  Abbott,  12.  http://manybooks.net/  support/  k/ 
kantimma /  kantimmaetext04ikfpml0iliad.pdf  (accessed  April  21,  2012). 

45  Immanuel  Kant,  Fundamental  Principals  of  the  Metaphysic  of  Morals,  6. 

46  Immanuel  Kant,  Fundamental  Principals  of  the  Metaphysic  of  Morals,  7. 


Kant  argues  that  humans  are  capable  of  understanding  practical 
pure  reason  but  are  “not  so  easily  able  to  make  it  effective  in  concreto  in 
his  life.”47  Humans  are  limited  in  reasoning  ability.  They  do  not 
comprehend  the  unconditional,  that  is,  morality  considered  out  of 
context,  necessity  of  the  moral  imperative.48  Kant’s  extreme  theoretical 
interpretation  of  morality  raises  questions  concerning  moral  imperatives 
as  rationalized  by  humans  and  ALEs.  Are  ALEs,  as  a  purer  form  of 
rational  actors,  more  capable  than  humans  of  understanding  moral 
imperatives? 

Shifting  moral  hazard 

ALE  use  highlights  two  shifts  of  moral  hazard.  The  first  concerns 
Just  War  Theory  and  the  second  addresses  a  more  fundamental 
theoretical  question  concerning  the  nature  of  war. 

In  43  BCE  Cicero  addressed  two  of  the  primary  principles  of  Just 
War.  He  acknowledged  the  just  use  of  force  as  a  last  resort  and  that  the 
only  excuse  “for  going  to  war  is  that  we  may  live  in  peace  unharmed.”49 
Saint  Tomas  Aquinas  codified  modern  Just  War  Theory  around  1274  CE. 
Just  War  Theory  is  generally  considered  from  two  vantages,  jus  ad  bellum 
(justification  to  go  to  war,  also  addressed  as  why  we  go  to  war)  and  jus  en 
bello  (just  conduct  in  war  or  how  we  conduct  war).  Likewise,  in  May 
1983,  US  Bishops  outlined  criteria  to  meet  jus  ad  bellum  and  one  for  jus 
en  bello.50  Philosophers  differ  on  the  composition  of  each  category  but 
the  intentions  run  consistent. 

The  first  shift  of  moral  hazard,  an  increased  willingness  to  go  to 
war  because  ALEs  promise  lessened  human  casualties  on  the  user’s  side, 

47  Immanuel  Kant,  Fundamental  Principals  of  the  Metaphysic  of  Morals,  8. 

48  Immanuel  Kant,  Fundamental  Principals  of  the  Metaphysic  of  Morals,  140. 

49  “The  Church’s  Tradition  of  Just  War,’’  http://www.catholicpeacefellowship.org/downloads/just_war.pdf 
(accessed  January  2012). 

50  “The  Church’s  Tradition  of  Just  War,”  2.  Referencing,  US  Bishops,  The  Challenge  of  Peace  May  3, 

1983,  http://old.usccb.org/sdwp/international/TheChallengeofPeace.pdf  18-22  (accessed  May  9,  2012),  18- 
22. 


concerns  the  jus  en  bello  principle.  None  of  the  seven  jus  ad  bellum 
principles  outlined  by  the  Catholic  Church  are  explicitly  violated  through 
the  use  of  ALEs.51  However  reducing  the  threshold  of  pain,  through  the 
use  of  ALEs,  may  alter  the  weighted  perception  of  Just  War  criteria 
resulting  in  a  greater  willingness  to  go  to  war.  Section  3.2.1  addresses 
the  concern  that  ease  of  killing  expands  willingness  to  kill  or  lowers  the 
threshold  for  proportionality  or  discrimination  in  killing. 

Understanding  the  nature  of  war  versus  the  character  of  war  is 
critical  to  an  understanding  of  the  affects  created  by  ALE  systems. 
Clausewitz  defined  the  nature  of  war  as  enduring  while  the  character  and 
conduct  of  war  changed  based  on  the  nature  of  societies  as  well  as  the 
times  and  prevailing  conditions.  Societal  conditions  drive  the  character  of 
war  as  well  as  the  perception  of  moral  hazard.  Delineating  between 
changed  character  and  changed  nature  of  war  addresses  the  perception 
or  realized  negation  of  moral  hazard. 

The  following  amicus  curiae  brief  addresses  concerns  of  moral 
repercussions  and  the  decline  of  military  ethos: 


Your  Honor 

Thank  you  for  the  invitation  to  address  the  subject  of 
shifting  moral  hazards  and  ALEs.  I  understand  the  focus  of 
the  trial  at  hand  surrounds  accountability  of  actions  and  the 
legal  standing  of  a  machine.  I  wrote  against  the  use  of  ALEs 
while  attending  the  Air  War  College.  My  focus  at  the  time 
was  on  how  ALEs  could  change  the  character  and  possibly 
the  nature  of  war  and  affect  the  military  ethos.  My 

51  “The  Church’s  Tradition  of  Just  War.”  The  US  Bishops  outlined  the  following  seven  criteria  for  a  just 
war:  just  cause,  competent  authority,  right  intention,  last  resort,  probability  of  success,  proportionality,  and 
comparative  justice.  An  eight  criteria  of  discrimination  was  placed  on  the  conduct  of  soldiers  in  war. 


underlying  arguments  rested  on  the  erosion  of  just  war 
principles  related  to  the  willingness  to  go  to  war  as  well  as 
the  changing  character  of  war. 

While  these  topics  may  not  be  driving  considerations 
in  your  decision  to  assess  legal  standing,  I  ask  that  you  keep 
these  arguments  in  the  forefront  of  your  mind  to  ensure  a 
whole  picture  view  of  the  consequences  of  ALE  technology. 
Applying  legal  standing  to  a  machine  will  increase  the  moral 
distance  between  political  decision  makers  and  warriors  and 
the  consequences  of  our  collective  actions. 

I  do  not  dispute  the  possibility  of  an  autonomous 
“computer  system  which  solely  relies  upon  its  own  internal 
programming  and  capabilities  to  conduct  and  execute  all 
elements  of  the  kill  chain.  Programmed  with  Rules  of 
Engagement  (ROEs),  Law  of  Armed  Conflict  (LOAC), 
conventions,  and  heuristics,  and  ethical  black  box  within  the 
ALE  system  [that]  independently  evaluates  each  step  of  the 
kill  chain  and  decides  when  and  whether  to  engage  an 
intended  target  with  lethal  force.”52  I  even  consider  the  use  of 
ALEs  as  amoral  in  that  morality  does  not  reside  in  the 
means  of  killing.  However,  the  act  of  an  ALE  killing,  is 
“contrary  to  military  ethics  and  detrimental  to  the  military 
profession  and  thus  should  be  prohibited.”53 

Our  willingness  to  go  to  war  is  based  on  a  formula  that 
the  damage  inflicted  plus  the  costs  incurred  must  be 
approximate  to  the  good  expected.54  ALE  technology 
threatens  jus  ad  bellum  arguments  because  ALEs  “limit  the 


52  Michael  R.  Contratto,  “The  Decline  of  Military  Ethos  and  the  Profession  of  Arms:  An  Argument  Against 
Autonomous  Lethal  Engagements,”  (submitted  in  partial  fulfillment  of  requirements  of  the  Air  War 
College,  Air  University,  February  201 1),  2. 

53  Contratto,  “Decline  of  Military  Ethos  and  the  Profession  of  Arms”  2. 

54  Contratto,  “Decline  of  Military  Ethos  and  the  Profession  of  Arms”  5  (footnote). 


risk  to  nations  and  their  soldiers  to  such  an  extent  that  it 
significantly  removes  political  barriers  and  lowers  the  bar  for 
proper  authorities  to  declare  and  enter  war.  The  nuanced 
argument  maintains  that  ALE  would  skew  the  arithmetic  of 
proportionality  such  that  force  may  not  be  used  as  a  last 
resort... The  opposing  perspective  is  that  from  the  tip  of  the 
phalanx,  to  a  sniper  at  several  hundred  meters,  to  an 
unmanned  aerial  system  operator  sipping  his  coffee  at 
Creech  Air  Force  Base  -  isn’t  ALE  simply  the  next 
evolutionary  step  in  the  pursuit  of  risk  free  war?  For  the 
military,  the  full  impact  of  this  question  is  better  evaluated 
in  terms  of  the  overall  context  of  the  profession  of  arms  and 
military  ethos  rather  than  jus  ad  bellum.”55 

The  formula  relating  inflicted  damage,  incurred  costs, 
and  good  expected,  also  links  the  rise  of  ALEs  to  the 
changing  character,  and  possibly,  the  nature  of  war.  The 
nature  of  war  changes  when  ALEs  replace  human  endeavors. 
“If  robots  accomplish  our  most  dangerous  and  near 
impossible  missions  what  then  becomes  the  role  and 
purpose  of  the  soldier?”56 

The  Clausewitzian  nature  of  war  is  enduring  so  long  as 
the  human  enterprise  promulgates  warfare.57  “If  professions 
are  ‘quintessentially’  human  institutions  operating  in  areas 
‘where  humanity’s  most  profound  concerns  arise’  and  we 
willingly  relinquish  the  responsibility  for  the  toughest 
decisions  in  our  sphere  of  expertise,”  what  becomes  of  the 
nature  of  war?58 


55  Contratto,  “Decline  of  Military  Ethos  and  the  Profession  of  Arms,”  4,6. 

56  Richard  Swain,  “Reflection  on  an  Ethic  of  Officership,”  Parameters  38,  no.  1  (Spring  2007),  15. 

57  This  thesis  author’s  interpretation  of  the  difference  between  a  character  and  nature  of  warfare. 

58  Contratto,  “Decline  of  Military  Ethos  and  the  Profession  of  Arms,”  15.  Contratto  quotes  Don  M.  Snider, 
Dissent  and  Strategic  Leadership  of  the  Military  Professions,  February  2008,  Strategic  Studies  Institute 


I  do  not  argue  against  the  justification  for  legal 
standing  of  an  ALE.  I  argue  for  the  moral  repercussions  of 
such  decision.  We  are  faced  with  “challenging  ethical 
dilemmas  as  technology  forces  [our]  professions  to  decide  if 
they  are  willing  to  relinquish  their  moral  agency  to 
autonomous  agents.  I  maintain  that  relinquishing  the  moral 
responsibility  for  the  endeavors  that  make  us  most  human 
does  not  speak  well  for  humanity.”59  If  you  assign  moral 
agency  or  legal  standing  to  this  ALE  you  will  remove  levels  of 
morality  and  ethical  responsibility  from  all  professions 
reliant  on  autonomous  systems  to  include  the  medical,  legal, 
and  military.60 


Respectfully  yours, 

Michael  R.  Contratto,  Lt  Col,  USAF61 

Shifts  willingness  to  go  to  war? 

Reuters  reported  in  January  2012  that  President  Obama 
authorized  “five  times  as  many  drone  strikes  as  George  W.  Bush 
authorized  in  his  second  term  in  the  White  House,”  raising  the  Pakistani 
death  toll  due  to  strikes  from  dozens  a  year  to  over  a  thousand  in  20 10. 62 
In  2011,  reports  of  American  drone  strikes  spread  to  Somalia  and  Libya. 
Rohde  attributes  the  increase  and  spread  of  American  drone  attacks  as  a 
means  for  “cash-strapped  officials”  to  “eliminate  the  United  States’ 


online  article  at  http://www.strategicstudiesinstitute.army.mil/pdffiles/pub849.pdf,  12.  Contratto’s  focus 
was  on  the  erosion  of  professional  institutions  not  on  the  changing  nature  of  warfare.  However  his 
argument  for  the  breakdown  of  professions  and  the  changing  nature  of  warfare  both  rest  on  the  removal  of 
humanity  from  the  profession  of  arms. 

59  Contratto,  “Decline  of  Military  Ethos  and  the  Profession  of  Arms,”  19-20. 

60  Contratto,  “Decline  of  Military  Ethos  and  the  Profession  of  Arms,”  19-20. 

61  The  thesis  author  used  artistic  license  to  represent  Lt  Col  Contratto’s  argument. 

62  David  Rohde,  “Reuters  Magazine:  The  drone  wars,”  Business  and  Financial  News,  Breaking  US  & 
International  News  on  Reuters.com ,  (January  26,  2012)  http://www.  reuters.  com/  article/  2012/  01/  26/  us- 
david-  rohde-  drone-  wars-  idUSTRE80Pl  1120120126  (accessed  on  March  21,  2012). 


enemies  at  little  human,  political,  or  financial  cost.”63  Rohde’s  report 
highlights  a  concern  that  autonomous  or  remotely  controlled  systems 
increase  both  the  willingness  to  go  to  war  as  well  as  the  willingness  to 
stay  at  war.  The  concern  is  not  relevant  to  the  legal  standing  argument 
of  the  ALE  but  it  is  a  central  concern  for  the  responsibility  and 
accountability  of  ALE  development  and  use.  Is  it  possible  for  a  society  to 
employ  an  ALE  without  easing  the  burden  of  killing  and  lowering  the 
threshold  for  proportionality  or  discriminate  killing? 

Operation  United  Protector  is  an  example  of  political  desire  to 
perpetrate  violent  action  while  minimizing  risk  to  human  and  political 
capital.64  In  essence,  a  no  fly  zone  is  a  method  of  exerting  violent 
measures  while  minimizing  hurt  on  the  side  of  the  executing  nation.  A 
small  logical  leap  expands  the  reduction  in  hurt  with  the  application  of 
remotely  controlled  systems  such  as  the  USAF’s  RPVs  or  the  USA’s  Big 
Dog.  The  logic  follows  that  use  of  autonomous  systems  could  decrease 
the  hurt  variable  while  also  lowering  empathy.  Without  empathy  for  the 
enemy  or  the  potential  for  hurt,  Clausewitz’s  violence,  chance,  and 
reason  could  become  skewed  with  reason  diminishing  and  chance 
morphing  into  a  non-human  driven  variable. 

Has  the  nature  of  war  changed?  Redefining  War? 

Near  the  end  of  book  one  in  On  War,  Clausewitz  summarizes  the 
nature  and  character  of  war.  He  cautions  readers  that  “wars  must  vary 
with  the  nature  of  their  motives  and  of  the  situation  which  give  rise  to 
them.”65  He  states  that  the  most  supreme  act  of  judgment  required  of 


63  Rohde,  “Reuters  Magazine:  The  drone  wars.” 

64 

Lt  Gen  Ralph  J  Jodice  II,  OUP  CFACC,  Op  Unified  Protector  (OUP)  Mission  Brief  as  presented  to  the 
School  of  Advanced  Air  and  Space  Studies  (February  2012). 

65  Carl  Von  Clausewitz,  On  War ,  ed.  and  trans.  Michael  Howard  and  Peter  Paret  (Princeton,  NJ:  Princeton 
University  Press,  1976),  88. 


decision  makers  is  to  recognize,  first,  the  character  of  war  and  warns 
against  turning  it  into  “something  that  is  alien  to  its  nature.”66 

After  acknowledging  that  the  character  for  each  war  is  situationally 
dependent,  Clausewitz  summarizes  the  nature  of  war  as  conditional  on 
“three  tendencies  [that]  are  like  three  different  codes  of  law,  deep-rooted 
in  their  subject  and  yet  variable  in  their  relationship  to  one  another.”67 
The  three  tendencies  are  1)  blind  natural  force,  the  primordial  violence 
that  mainly  concerns  the  people,  2)  play  of  chance  and  probability  within 
which  the  creative  spirit  of  the  commander  and  military  are  free  to  roam, 
and  3)  reason,  drawn  from  the  government  as  an  instrument  of  policy.68 
Therefore,  so  long  as  wars  are  composed  of  these  three  elements;  force, 
chance,  and  reason,  the  nature  of  war  is  enduring.  Regarding  ALEs,  the 
questions  remains,  does  the  addition  of  autonomous  lethal  systems  into 
war  change  the  elements,  thereby  altering  the  nature  of  war,  or  is  this 
simply  a  case  of  understanding  a  different  situation  and  thus  a  changed 
character  of  war?  Analysis  requires  an  examination  of  each  element 
separately. 

The  first  element,  blind  natural  force,  derives  the  passion  of  war 
from  the  primordial  violence  of  a  populous.  Without  a  check,  this 
violence  will  spiral  towards  theoretical  absolute  forms  of  war  ending  in 
the  annihilation  of  at  least  one  side.  Passion  of  violent  force  affects  war 
prior  to  the  outbreak.  It  also  serves  as  an  endurance  fuel  for  continued 
execution.  This  element  is  commonly  known  as  will  of  the  people. 

At  the  juncture  in  time  of  the  Machine  on  Trial,  autonomous 
systems  execute  aspects  of  war  through  the  application  of  force.  They  are 

66  Von  Clausewitz,  On  War,  89 

67  Von  Clausewitz,  On  War,  89. 

68  Von  Clausewitz,  On  War,  89.  Von  Clausewitz  dedicates  “Book  One”  of  his  On  War,  to  theoretical 
questions  defining  war.  He  switches  between  addressing  war  in  the  absolute  theoretical  form  where  both 
reason  and  chance  are  restricted  allowing  absolute  force  to  drive  war  towards  annihilation  and  a  politically 
restricted  limited  version  of  reality  where  both  chance  and  reason  allow  time  and  other  considerations  to 
limit  war.  He  states  on  page  8 1  “Thus  it  follows  that  without  any  inconsistency  wars  can  have  all  degrees  of 
importance  and  intensity,  ranging  from  a  war  of  extermination  down  to  simple  armed  observation.”  Along 
these  varying  degrees  it  is  the  character,  not  the  nature  of  war,  that  changes. 


not  the  instigators  of  blind  natural  force  therefore  ALEs  do  not  contribute 
to  nor  alter  the  human  driven  tendency  of  violence  prior  to  the  outbreak 
of  war.  Since  ALEs  execute  in  war  they  must  alter  the  element  of  blind 
force  in  some  manner.  By  reducing  the  number  of  humans  involved  in 
the  conflict,  it  is  reasonable  to  state  that  ALEs  reduce  the  pain  felt  by  the 
ALE  supply  State.  State  assets  are  still  risked,  therefore,  some  pain  still 
feeds  back  to  the  populous  affecting  the  will  of  the  people.  ALEs  do  not 
change  the  basic  nature  of  war  through  the  first  element. 

A  complimentary  argument  supports  ALE  or  stand-off  weapons  use 
as  a  means  of  increasing  the  capacity  for  restraint  in  war.69  Clausewitz 
describes  the  tendencies  of  war  to  spiral  into  theoretical  total  war  if  the 
passion  for  violence  is  not  restrained  by  reason.  By  decreasing  the  hurt 
felt  by  a  nation  employing  ALEs,  an  argument  exists  that  the  passion  for 
continued  violence  lessens  allowing  for  a  greater  capacity  for  reason  and 
thus  restraint  in  war. 

The  second  element  of  chance  relates  to  the  creative  spirit  and 
decisions  of  the  military  members  across  varying  levels  of  conflict.  ALE 
actions  affect,  but  do  not  drive,  strategic  and  operational  levels  of 
command.  Unless  the  ALEs  intentionally  link  the  political,  strategic,  and 
operational  plans  of  war,  the  ALEs  do  not  affect  the  element  of  chance 
any  more  so  than  a  foot  soldier — that  is,  above  the  tactical  level.70  ALE 
actions  are  situationally  driven.  They  adapt  to  tactical  situations. 
Therefore,  ALEs  are  relegated  to  define  the  character,  not  nature  of  war. 
ALEs  do  not  change  the  basic  nature  of  war  through  the  second  element. 


69  Dr.  Everett  Dolman  raised  the  complimentary  point  during  the  thesis  feedback  process.  He  stated  “the 
flip  side  of  reducing  pain,  thus  increasing  willingness  to  use  force,  is  the  notion  that  since  I  am  not  feeling 
pain,  I  am  not  dominated  by  my  passion  to  return  fire  upon  those  shooting  at  me  or  get  revenge  for  a  loved 
one  lost,  ALEs  or  stand-off  weapons  can  increase  the  capacity  for  restraint  in  war.” 

70  The  thesis  author  acknowledges  that  an  Army  Corporal  can  affect  strategic  policy  through  tactical  actions 
but  does  not  agree  that  this  link  makes  for  a  “strategic  corporal.”  The  difference  lies  with  intent  and 
replicated  actions.  This  same  basis  is  used  to  link  ALE  tactical,  possibly  even  operational  plans,  to  strategic 
effect  without  allowing  for  ALE  strategic  consideration. 


Reason,  derived  from  needs  of  the  government  that  exceed 
annihilation  of  opponents,  delineates  the  third  nature  of  war  element.  As 
with  the  first  and  second  elements,  unless  ALEs  are  the  decision  makers 
for  going  to  or  ending  war,  ALEs  do  not  affect  this  element.  Therefore,  so 
long  as  humans  cause  war  and  decide  when  to  start  and  end  wars,  ALEs 
do  not  change  the  nature  of  war,  only  the  character. 

The  character  of  war,  being  situationally  dependent,  defines  how 
we  fight  as  well  as  who  or  what,  not  why  we  fight.  Why  we  fight  speaks  to 
the  nature  of  war.  The  nature  of  war  is  as  enduring  as  the  nature  of 
humanity.  Clausewitz  points  out,  for  example,  that  the  art  of  war 
changed  in  the  last  decade  of  the  eighteenth  century.  The  blind  force  of 
the  people  affected  the  political  reasoning  of  the  government  resulting  in 
the  French  Revolution.  Failure  to  identify  the  enduring  nature  of  war,  but 
changing  character  of  the  French  government,  cost  other  European 
nations  primacy  in  war.  “Transformation  of  the  art  of  war  resulted  from 
the  transformation  of  politics  . . .  War  is  an  instrument  of  policy.  It  must 
necessarily  bear  the  character  of  policy  and  measure  by  its  standards.”71 
Until  ALEs  intentionally  create  policy,  ALEs  do  not  affect  the  nature  of 
war. 


71  Von  Clausewitz,  On  War ,  609-610. 


Chapter  4:  Historical  Legal  Review 


The  life  of  the  law  has  not  been  logic:  it  has  been  experience  ... 
The  law  embodies  the  story  of  a  nation's  development  through 
many  centuries ,  and  it  cannot  be  dealt  with  as  if  it  contained 
only  the  axioms  and  corollaries  of  a  book  of  mathematics.  In 
order  to  know  what  it  is,  we  must  know  what  it  has  been,  and 
what  it  tends  to  become.  We  must  alternately  consult  history 
and  existing  theories  of  legislation.  But  the  most  difficult  labor 
will  be  to  understand  the  combination  of  the  two  into  new 
products  at  every  stage.  The  substance  of  the  law  at  any 
given  time  pretty  nearly  corresponds,  so  far  as  it  goes,  with 
what  is  then  understood  to  be  convenient;  but  its  form  and 
machinery,  and  the  degree  to  which  it  is  able  to  work  out 
desired  results,  depend  very  much  upon  its  past. 

-  Justice  Oliver  Wendell  Holmes 
Treaty  on  Common  Law 


The  basis  of  law  formation  is  declaration  and  precedence.  Without 
clear  guidance,  hearings  such  as  the  Machine  on  Trial  will  be  argued  on 
analogous  precedence.  Four  case  studies  follow  as  analogies  for  key 
arguments  in  Machine  on  Trial. 

The  recent  US  Supreme  Court  ruling  on  the  rights  of  corporations 
represents  movement  of  society  towards  recognizing  the  roles  and 
protection  of  non-sentient  entities,  as  legal  persons,  along  the  lines 
formerly  limited  to  humans.  A  second  case  argues  for  legal  standing 
based  on  moral  agency.  Following  arguments  for  personhood  or  legal 
standing,  the  mantrap  and  landmine  cases  represent  a  sliding  scale  of 
acceptable  discrimination  for  existing  autonomous  systems.  The  mantrap 
case  requires  increased  discriminatory  capability  as  well  as  flexibly  for 
use  of  the  autonomous  system  (only  acceptable  with  physical  occupation 
of  a  house  or  base).  The  US  Government  refuses  to  sign  anti-land  mine 
treaties.  This  indicates  its  willingness  to  maintain  a  non-discriminatory 
autonomous  system  for  use  in  future  conflicts. 


For  some  purposes,  corporations  are  persons. 

-  Retired  Supreme  Court  Justice 
John  Paul  Stevens 

Defining  Persons  Analogies:  Corporation  Rights 

The  fate  of  the  ALE  rested  upon  agreement  of  one  point.  For  the 
ALE  to  stand  trial,  able  to  be  held  accountable  for  actions,  the  ALE  must 
hold  standing  in  the  eyes  of  the  court.  Such  standing  ensured  a  level  of 
rights  and  responsibilities  due  to  and  from  the  ALE.  One  means  of 
holding  standing  with  the  court  is  to  have  recognition  as  a  person. 1 
Recognition  as  a  person  with  some  level  of  rights  should  be  enough  to 
force  a  court  decision  in  favor  of  granting  standing  and  require  the 
Plaintiff,  or  prosecutor  in  a  criminal  case,  to  prove  their  case  against  the 
ALE  prior  to  a  final  verdict. 

The  ALE  representative  required  an  analogy  to  establish 
personhood  and  gain  standing  for  the  ALE.  The  representative  knew  that 
it  was  too  great  a  leap  to  challenge  societal  acceptance  of  the  ALE  as  an 
equal  person,  however,  precedent  existed  to  support  a  claim  that 
different  types  of  persons  had  different  rights.2  He  needed  to  establish 
the  ALE  as  a  type  of  person.  Without  closing  the  door  on  sentience,  the 


1  The  issue  of  standing  is  typically  examined  by  looking  at  whether  or  not  the  Plaintiff  in  a  case  has  a 
personal  stake  in  the  outcome  of  a  case.  Normally  the  standing  of  the  Defendant  is  assumed,  because  the 
Defendant  will  suffer  the  direct  consequence  of  the  court’s  decision.  There  are,  however,  some  instances 
where  it  is  not  clear  as  to  whether  or  not  a  Defendant  has  sufficient  standing.  A  recent  article  by  Professor 
Mathew  I.  Hall  addresses  the  issue  of  the  Defendant’s  standing  and  serves  as  a  departure  point  for  raising 
the  issue  of  whether  or  not  an  ALE  would  have  a  sufficient  stake  in  the  outcome  of  a  trial  so  as  to  satisfy 
the  constitutional  requirement  for  a  party  to  have  standing.  See  Matthew  I.  Hall,  Standing  of  Intervenor- 
Defendants  in  Public  Law  Litigation,  80  Fordham  L.  Rev.  1539  (2012), 
http://ir.lawnet.fordham.edU/flr/vol80/iss4/2  (accessed  on  18  May  2012) 

2  Citizens  United  v.  FEC,  130,  S.  Ct.  Slip  Op  876  (2010),  available  at 

http://www.supremecourt.gov/opinions/09pdf/08-205.pdf.  (accessed  January  2012).  Justice  Stevens 
highlights  this  fact  in  his  dissenting  opinion  of  the  Citizens  United  case  heard  in  2010.  Additional  support 
for  this  claim  comes  from  the  constitutional  law  differentiating  natural  born  US  citizens  from  naturalized 
US  citizens  and  the  associated  rights  to  run  for  Presidential  office.  US  rights  are  also  differentiated  by  age 
(children  are  limited  in  rights  until  the  age  of  maturity)  and  felony  status. 


representative  sought  a  means  to  establish  non-sentient  personhood 
rights. 

In  2010  the  US  Supreme  Court  ruled  in  Citizens  United  v.  FEC  that 
First  Amendment  rights  of  corporations  are  protected.3  The  logic 
supporting  this  argument  is  the  link  that  people  and  groups  of  people  are 
protected  under  the  First  Amendment,  therefore  corporations  are  also 
protected.  The  implication  is  that  corporations  have  legal  standing  as 
persons  with  constitutional  protections.  The  ruling  established  no 
limitation  to  size,  management,  or  liability  requirements  for  the 
corporations.  Moreover,  the  ruling  solidified  precedence  for  the  protection 
of  non-sentient  entity  rights. 

Popular  interpretation  of  the  ruling  resulted  in  opposing  calls  that 
“Corporations  are  not  people.  Democracy  is  for  people.”4  A  significant 
portion  of  the  resentment  towards  this  ruling  stemmed  from  the 
apparent  inequality  of  rights  granted  to  corporate  persons  over  individual 
persons  concerning  electoral  campaign  influence.  Citizens  United  reduced 
the  burdensome  requirements  for  corporations  to  form  Super  Political 
Action  Committees  (PAC).  Advocates  viewed  this  as  a  loophole  within 
campaign  finance  reform  laws  and  limitations  of  undue  influence.  Super 
PACs,  backed  by  individuals  or  companies,  whose  financial  gains  were 
made  off  the  economic  system  and  whose  economic  power  far  exceeds 
the  average  person  of  the  state,  have  no  limitations  as  to  the  influence 
gained  from  free  speech  paid  for  by  Super  PAC  funding.  Super  PACs  are 
viewed  as  both  a  corrective  step  designed  to  allow  equal  opportunity  of 
free  speech  regardless  of  economic  position,  as  well  as  a  means  of 
allocating  greater  rights  to  the  wealthy  and  economically  strong 
corporations  over  the  average  citizens.  Opponents  of  the  Citizens  United 
decision  question  if  this  ruling  allows  economically  wealthy  corporations 

3  Citizens  United  v.  FEC,  130,  S.  Ct.  Slip  Op  876  (2010),  http://www.scotusblog.com/wp- 
content/uploads/2010/01/citizens-opinion.pdf  (accessed  18  May  2012) 

4  A  People’s  Movement,  blog,  http://democracyisforpeople.org/  (accessed  January  8,  2012). 


to  have  more  rights  than  the  average  citizen  by  allowing  unequal  (based 
on  economic  conditions)  political  influence  through  free  speech.  The 
point  may  be  moot  as  the  US  Supreme  Court  has  ruled  that  not  all 
persons  have  the  same  rights. 

The  precedent  of  corporations  as  persons  existed  in  state  and 
federal  law  prior  to  the  most  recent  US  Supreme  Court  ruling.  An  1818 
case,  Trustees  of  Dartmouth  College  v.  Woodward ,  affirmed  the 
application  of  the  Contracts  Clause  of  the  Constitution  to  private 
corporations,  suggesting  that  corporations  had  certain  rights.5  The  first 
claim  of  constitutional  rights,  the  claim  for  5th  Amendment  rights,  of  a 
corporation  occurred  in  1893  with  the  holding  in  Noble  v.  Union  River 
Logging  Railroad  Company .  In  Noble  the  Court  held  that  the  Government 
action  fell  under  the  Fifth  Amendment  and  entitled  the  Union  River 
Logging  Railroad  Company  to  due  process.6  The  1970  case  of  Ross  v. 
Bernhard  established  the  right  of  corporations  to  a  jury  trial,  under  the 
Seventh  Amendment.7  Several  court  rulings  in  the  1970s  firmed  the 
precedent  of  constitutional  amendment  protection  of  corporations.8 
Each  of  these  cases  supported  the  view  that  even  a  non-sentient  entity 


5  Trustees  of  Dartmouth  College  v.  Woodward,  17  U.S.  (4  Wheat.)  518  at  654  (1819),  “[o]n  general 
principles,  that  in  these  private  eleemosynary  institutions,  the  body  corporate,  as  possessing  the  whole  legal 
and  equitable  interest,  and  completely  representing  the  donors,  for  the  purpose  of  executing  the  trust,  has 
rights  which  are  protected  by  the  constitution.”  17  U.S.  (4  Wheat.)  518,  at  654  (1819).  See  also  Reclaim 
Democracy  Blog ,  http://reclaimdemocracy.0rg/personhood/#signif1cant  (January  8,  2012). 

6  Noble  v.  Union  River  Logging  R.  Co.  147  U.S.  165,  174  (1893)  “revocation  of  the  approval  of  the 
secretary  of  the  interior,  however,  by  his  successor  in  office,  was  an  attempt  to  deprive  the  plaintiff  of  its 
property  without  due  process  of  law,  and  was,  therefore,  void.” 

(http://caselaw.lp. fmdlaw.com/scripts/getcase.pl?court=US&vol=147&invol=l 65  accessed  18  May  2012). 

7  Ross  v.  Bernhard ,  396  U.S.  531,  (1970)  “the  right  to  jury  trial  attaches  to  those  issues  in  derivative  actions 
as  to  which  the  corporation,  if  it  had  been  suing  in  its  own  right,  would  have  been  entitled  to  a  jury.”  at  533. 

8  Following  the  1970  ruling  supporting  the  right  to  jury  trial,  constitutional  amendment  rights  for 
corporations  were  held  up  in  several  cases.  U.S.  v.  Martin  Linen  Supply  Co .,  430  U.S.  564  (1976), 
established  5th  amendment  protection  from  double  jeopardy.  In  1978,  Marshall  v.  Barlow’s  Inc.  436  U.S. 
347  (1978),  safety  inspectors  were  required  to  obtain  search  warrants  under  the  4th  Amendment.  Also  in 
1978,  First  Nat.  Bank  of  Boston  v.  Bellotti  established  the  right  of  a  corporation  to  spend  money  intended  to 
influence  politics  when  a  Massachusetts  state  law  preventing  corporate  spending  to  influence  elections  was 
struck  down  First  Nat.  Bank  of  Boston  v.  Bellotti ,  435  U.  S.  765,  778  (1978).  Notably  one  of  the  most 
significant  decisions  came  from  the  Court  in  the  2010  Citizens  United  v.  FEC.  In  a  5-4  vote  the  Justices 
struck  down  their  previous  holding  in  Austin  v.  Michigan  Chamber  of  Commerce  which  had  limited 
corporate  political  contributions.  Austin  v.  Michigan  Chamber  of  Commerce ,  494  U.  S.  652  (1990). 


such  as  a  corporation  was  entitled  to  the  basic  protections  of  a  person 
under  the  Constitution  and  have  an  interest  in  the  outcome  of  a  case  or 
controversy. 

The  concept  of  corporation  personhood  is  further  evident  with  the 
ruling  in  Citizens  United.  The  Court  invalidated  restrictions  under  2 
U.S.C.  §  441(b)  because  it  banned  campaign  financial  expenditures  from 
individuals,  unions,  and  corporations,  because  the  Court  interpreted  this 
ban  as  impermissible  under  the  First  Amendment.  Section  44 1  (b)  was 
not  deemed  invalid  because  it  affected  the  rights  of  the  individual,  rather 
it  was  deemed  invalid  because  it  infringed  on  the  First  Amendment 
protection  of  free  speech  and  “fail[ed]  to  serve  any  substantial 
governmental  interest  in  stemming  the  reality  or  appearance  of 
corruption  in  the  electoral  process.”9 

The  pivotal  point  of  Citizens  United  stems  from  the  continued 
acknowledgement  and  logic  behind  personhood  status  of  corporations. 
The  Court  “rejected  the  argument  that  political  speech  of  corporations  or 
other  associations  should  be  treated  differently  under  the  First 
Amendment  simply  because  such  associations  are  not  “natural 
persons.”10  Corporations  are  persons  of  associations  with  a  “corporate 
form.”* 11  The  decision  in  Citizens  United  upheld  the  the  Court’s  previous 
ruling  in  Buckley  and  Bellotti  that  stated  “the  First  Amendment  principle 
that  the  Government  cannot  restrict  political  speech  based  on  the 
speaker’s  corporate  identity.”12  The  logic  inferred  behind  these 
statements  is  that  the  requirement  to  protect  the  rights  of  individual 
persons  and  associations  of  persons  is  greater  than  the  need  to  define 

9  Citizens  United  v.  FEC,  130,  S.  Ct.  Slip  Op  876  (2010),  citing  Buckley  v.  Valeo,  424  U.S.  1  (1976)  at  47- 
48. 

10  Citizens  United  v.  FEC,  130,  S.  Ct.  Slip  Op  876  (2010),  citing  First  Nat.  Bank  of  Boston  v.  Belloti  435 
U.S. 765  (1978)  at  776. 

11  Citizens  United  v.  FEC,  130,  S.  Ct.  Slip  Op  876  (2010).  In  discussing  the  Court’s  decision  to  strike 
down  Austin  it  stated  “If  the  anti  distortion  rationale  were  to  be  accepted,  however,  it  would  permit 
Government  to  ban  political  speech  simply  because  the  speaker  is  an  association  that  has  taken  on  the 
corporate  form.” 

12  Citizens  United  v.  FEC,  130,  S.  Ct.  Slip  Op  876  (2010). 


the  parameters  of  when  personhood  ends.  The  Court  opted  to  consider 
all  corporations  as  persons  with  First  Amendment  rights  rather  than 
break  out  for-profit,  non-profit,  or  limited  liability  corporations.  Justice 
Kennedy  stated  in  the  majority  opinion  that  “We  decline  to  adopt  an 
interpretation  that  requires  intricate  case-by-case  determinations  to 
verify  whether  political  speech  is  banned,  especially  if  we  are  convinced 
that,  in  the  end,  this  corporation  has  a  constitutional  right  to  speak  on 
this  subject.”13 

According  to  The  Entrepreneur,  “a  corporation  is  a  body  -  it  is  a 
legal  person  in  the  eyes  of  the  law.  It  can  bring  lawsuits,  can  buy  and  sell 
property,  contract,  be  taxed,  and  even  commit  crimes.  Its  most  notable 
feature:  a  corporation  protects  its  owners  from  personal  liability  for 
corporate  debts  and  obligations  -  within  limits...  The  most  common 
motivation  for  incurring  the  cost  of  setting  up  a  corporation  is  the 
recognition  that  the  shareholder  is  not  legally  liable  for  the  actions  of  the 
corporation.  This  is  because  the  corporation  has  its  own  separate 

existence  wholly  apart  from  those  who  run  it.”14  The  recognition  that  a 
corporation  exists  outside  the  liability  and  responsibility  of  the 
shareholders  or  the  persons  who  created  it  is  seems  contrary  to  the 
intention  of  the  Supreme  Court  interpretation  of  corporations  as  persons. 
Since  the  Court  cannot  differentiate  corporations  closely  associated  with 
persons  or  association  of  persons,  the  Court  has  defaulted  to  categorizing 
all  corporations  as  persons  based  on  the  connotation  of  associations  of 
persons. 

Considering  the  validity  of  a  corporation’s  personhood  status  even 
though  it  was  created  to  ensure  “its  own  separate  existence  wholly  apart 
from  those  who  run  it,”  personhood  must  be  based  on  something  other 


13  Citizens  United  v.  FEC,  130,  S.  Ct.  Slip  Op  876  (2010). 

14  “Incorporating  Your  Business,”  Entrepreneur.com,  http://www.entrepreneur.com/article/77730  (accessed 
February  2,  2012). 


than  an  association  with  people.15  If  constitutional  rights  extend  to 
corporations,  and  the  concept  of  constitutional  rights  extends  beyond  the 
rights  of  individuals,  then  a  level  of  responsibility  from  those  individuals 
is  inferred. 16  The  right  to  free  speech  does  not  exist  for  the  sole  purpose 
of  expressing  thoughts;  it  exists  to  ensure  people  have  the  ability  and 
thus  the  responsibility  to  speak  freely  concerning  the  actions  of  their 
government.  A  system  of  government  created  “of  the  people,  for  the 
people,  and  by  the  people,”  requires  action  from  the  body  of  people  to 
function. 17  This  action  implies  responsibility.  This  responsibility  is 
protected  so  long  as  the  person  meets  requirements  such  as  citizenship 
and  age.  However,  the  responsibility  is  partially  repealed  for  a  convicted 
felon. 18  A  felon,  by  demonstrating  a  lack  of  responsibly,  loses  a 
corresponding  portion  of  political  rights.  Therefore,  the  definition  of  a 
person  must  have  some  association  with  responsibility  to  society. 

The  ALE  in  question  was  created  to  secure  liberties,  enforce  laws, 
and  our  national  will.  In  doing  so,  the  ALE  demonstrated  decision 
capability  related  to  both  discrimination  and  proportionality.  If  it  is  an 
entity,  much  like  a  corporation,  that  takes  separate  action  “wholly  apart 
from  those  who  run  it,”  and  who’s  actions,  unlike  a  corporation,  exist  to 
demonstrate  responsibility  to  society,  shouldn’t  it  have  legal  standing?. 


15  “Incorporating  Your  Business,”  Entrepreneur.com,  http://www.entrepreneur.com/article/77730  (accessed 
February  2,  2012). 

16  Reclaim  Democracy  Blog,  http://reclaimdemocracy.0rg/personhood/#signif1cant  (January  8,  2012). 
Following  the  1970  ruling  supporting  the  right  to  jury  trial,  constitutional  amendment  rights  for 
corporations  were  held  up  in  several  cases. 

17  President  Abraham  Lincoln,  speech  to  dedicating  the  Gettysburg  military  cemetery,  http://www. 
whitehouse.  gov/  about/presidents/abrahamlincoln  (accessed  January  2012). 

18  Alan  Ellis  and  Peter  J.  Scherr,  “Federal  Felony  Conviction,  Collateral  Civil  Disabilities,”  The  Law  Office 
of  Allan  Ellis  (Fall  1996),  http://www.alanellis.com/CM/Publications/federal-felony-conviction.asp 
(accessed  February  2,  2012).  As  an  example,  in  Beecham  v.  United  States,  1 14  S.  Ct.  1669  (1994)  it  was 
held  that  even  if  the  felon's  civil  rights  have  been  restored  under  state  law,  federal  firearms  disabilities 
continue  to  apply  to  a  person  convicted  of  a  federal  offense.”  Per  the  referenced  site. 


Standing  and  Moral  Agent 

The  legality  aspect  of  the  main  thesis  rests  on  the  concept  of  legal 
standing.  Traditionally  personhood  is  required  for  legal  standing.  The 
following  amicus  brief  highlights  consideration  for  standing  based,  not  on 
personhood,  but  on  moral  agent. 

Dear  Sir, 

I  understand  you  are  considering  the  legal  standing  of 
an  ALE.  I  recognize  the  heavy  burden  of  your  decision  and 
offer  my  humble  perspective  on  the  concept  of  moral  agency 
as  a  means  of  lessoning  the  considerations  of  spirituality, 
soul,  consciousness,  life,  and  other  humanity  centric 
versions  of  agency.  It  is  my  opinion  that  “robots  are  moral 
agents  when  there  is  a  reasonable  level  of  abstraction  under 
which  we  must  grant  that  the  machine  has  autonomous 
intentions  and  responsibilities.  If  the  robot  can  be  seen  as 
autonomous  from  many  points  of  view,  then  the  machine  is 
a  robust  moral  agent.”19  It  is  also  my  opinion  that  all  moral 
agents  must  be  afforded  “corresponding  rights  and 
responsibilities.”20 

It  seems  to  me  that  the  heart  of  your  issue  is  whether 
or  not  the  machine  on  trial  is  morally  culpable.  I  believe 
there  are  four  considerations  required  to  establish  moral 
culpability.  An  entity  must  have  moral  value  to  be  morally 
culpable.  In  2006  I  argued  for  two  cases  that  place  moral 
value  on  guide  dogs.  “The  least  controversial  is  to  consider 
things  that  perform  their  function  well  have  a  moral  value 
equal  to  the  moral  value  of  the  actions  they  facilitate.  A  more 


19  John  P.  Sullins  “When  is  a  Robot  a  Moral  Agent?,”  International  Review  of  Information  Ethics  6, 
(December  2006):  29 

20  Sullins,  “When  is  a  Robot  a  Moral  Agent?”  29. 


contentious  claim  is  the  argument  that  animals  have  their 
own  wants,  desires  and  states  of  well  being,  and  this 
autonomy,  though  not  as  robust  as  that  of  humans,  is 
nonetheless  advanced  enough  to  give  the  dog  a  claim  for 
both  moral  rights  and  possibly  some  meager  moral 
responsibilities  as  well.”21  At  the  time  I  was  criticized  for 
comparing  a  robotic  tool  to  a  living  creature.  The  common 
view  of  autonomous  robots  in  2006  tended  towards  the 
telerobot  concept.  I  am  basing  my  argument,  not  on  a 
telerobot,  but  on  the  philosophical  subject  of  autonomous 
robots.22  With  moral  value  assigned,  the  three  remaining 
requirements  for  moral  agency  are:  1.  Significant  autonomy, 
2.  Intentional  behavior,  and  3.  Position  of  responsibility  to 
another  moral  agent. 

“‘Autonomy’  is  a  difficult  concept  to  pin  down 
philosophically.  I  mean  to  use  the  term  ‘autonomy’  in  the 
engineering  sense,  simply  that  the  machine  is  not  under  the 
control  of  any  other  agent  or  user.  If  this  autonomous  action 
is  effective  in  achieving  the  goals  and  tasks  of  the  robot,  then 
we  can  say  the  robot  has  effective  autonomy.  Autonomy  as 
described  is  not  sufficient  in  itself  to  ascribe  moral 
agency.”23  I  doubt  many  of  us  would  ascribe  moral  agency  to 
our  Roomba  vacuum  cleaners  though  the  robot  may  exceed 
the  autonomous  definition  as  described. 


21  Sullins,  “When  is  a  Robot  a  Moral  Agent?”  25. 

22  Sullins,  “When  is  a  Robot  a  Moral  Agent?”  25-26.  Sullins  defines  a  telerobot  as  “remotely  controlled 
machines  that  make  only  minimal  autonomous  decisions.”  The  operator  provides  the  intelligence  for  the 
machine  such  as  the  NASA  Mars  Rovers,  telerobotic  surgery,  telerobotic  nurses,  Predator  drones,  and  the 
Army  SWORD.  To  bypass  a  “full  discussion  of  the  meaning  of  ‘autonomy’”  as  a  philosophical  subject, 
Sullins  uses  the  roboticists  definition:  “autonomous  robots  must  be  capable  of  making  at  least  some  of  the 
major  decisions  about  their  actions  using  their  own  programing.”  This  definition  covers  a  robot  vacuum 
cleaner  deciding  on  a  vacuum  pattern  to  the  moral  and  ethical  reasoning  of  a  robotic  caregiver  deciding  the 
best  way  to  interact  with  a  patient. 

23  Sullins,  “When  is  a  Robot  a  Moral  Agent?”  28. 


The  second  requirement  of  intentionality  requires 
enough  complex  behavior  “that  one  is  forced  to  rely  on 
standard  folk  psychological  notions  of  predisposition  or 
‘intention’  to  do  good  or  harm.  There  is  no  requirement  that 
the  actions  really  are  intentional  in  a  philosophically 
rigorous  way,  nor  that  the  actions  are  derived  from  a  will 
that  is  free  on  all  levels  of  abstraction.  All  that  is  needed  is 
that  at  the  level  of  interaction  between  the  agents  involved, 
there  is  a  comparable  level  of  personal  intentionality  and  free 
will  between  all  the  agents.”24 

A  legal  discussion  of  autonomous  machines  starts 
when  the  machine  has  both  moral  value  and  intentionality. 
These  qualities,  without  a  complex  understanding  of  the 
robots  role  in  the  responsibility  to  another  moral  agent,  are 
enough  to  make  it  “deserving  of  moral  consideration, 
meaning  that  one  would  have  to  have  a  good  reason  to 
destroy  it  or  inhibit  its  actions,  but  we  would  not  be  required 
to  treat  it  as  a  moral  equal.”25  The  ALE  in  your  courtroom 
has  demonstrated  sufficient  autonomy.  Your  insight  and 
other  experts  must  clarify  if  the  intention  requirement  is 
met,  perhaps  by  executing  within  the  military  necessities  for 
target  discrimination.  Moral  equality,  based  on  an 
understanding  of  responsibility,  will  be  the  greater  challenge. 

The  responsibility  requirement  relies  on  the  concept  of 
belief.  “The  beliefs  do  not  have  to  be  real  beliefs,  they  can  be 
merely  apparent.  The  machine  may  have  no  claim  to 
consciousness,  for  instance,  or  a  soul,  a  mind,  or  any  of  the 
other  somewhat  philosophically  dubious  entities  we  ascribe 
to  human  specialness.  These  beliefs,  or  programs,  just  have 

24  Sullins,  “When  is  a  Robot  a  Moral  Agent?”  28. 

25  Sullins,  “When  is  a  Robot  a  Moral  Agent?”  29. 


to  be  motivational  in  solving  moral  questions  and 
conundrums  faced  by  the  machine.  [An  understanding  of 
responsibility  is  met]  when  the  robot  behaves  in  such  a  way 
that  we  can  only  make  sense  of  that  behavior  by  assuming  it 
has  a  responsibility  to  some  other  moral  agent  (s).” 

I  respectfully  submit  this  amicus  brief  for  your 
consideration. 

John  P.  Sulliins26 

Assistant  Professor,  Philosophy  Dept 

Sonoma  State  University 

Chapter  2  highlighted  the  legal  focus  on  discrimination  and 
proportionality.  Proportionality  deals  with  the  means  and  the  moderation 
of  force  with  respect  to  balancing  military  necessity  with  destruction. 
Assuming  the  means  are  capable  of  being  proportionate  (in  other  words  a 
thermonuclear  warhead  is  not  used  against  a  city  to  obtain  a  meager 
military  necessity)  then  proportionality  tends  to  be  examined  after  action. 
The  following  case  studies  focus  on  discrimination  since  discrimination 
first  requires  an  ability  to  discriminate  and  then  a  decision  process 
follows  to  consider  proportionate  response.  The  ALEs  ability  to  obtain 
standing  rests  more  on  a  capability  to  discriminate  than  on  the  moral  or 
legal  consequences  of  a  proportionate  result  from  the  action  in  question. 

Scale  of  Discrimination  Analogy:  Land  Mines  and  Mantraps 

The  US  currently  maintains  the  ability  to  use  autonomous  lethal 
systems.  The  minimum  use  of  discrimination  capability  does  not  reside 
with  the  development  of  an  autonomous  weapon  system  but  rather  with 
conditional  necessity  to  use  indiscriminate  weapons.  For  example,  anti¬ 
personnel  (AP)  landmines  represent  dumb  autonomous  lethal  systems. 
Once  they  are  fielded  AP  landmines  cannot  discriminate  based  on 


26  The  thesis  author  used  artistic  license  to  represent  Professor  Sulliins’  argument. 


identification  of  victim  nor  is  consideration  given  to  current  conditions 
and  proximity  of  non-targeted  entities.  AP  landmines  were  first  used  in 
large  numbers  during  the  Second  World  War.  They  were  designed  for 
area  denial  and  to  protect  anti-tank  mines.27  Canada’s  Foreign  Affairs 
and  International  Trade  organization  has  stated,  “Once  mines  are  laid, 
they  are  completely  indiscriminate  weapons.”28  The  indiscriminate 
nature  of  land  mines  spurred  the  1990s  International  Campaign  to  Ban 
Landmines  (ICBL).  In  1997  a  coalition  of  non-government  organizations, 
who  composed  the  ICBL,  was  awarded  the  Nobel  Peace  Prize  for  their  role 
in  establishing  the  Convention  on  the  Prohibition  of  the  Use,  Stockpiling, 
Production  and  Transfer  of  Anti-Personnel  Mines  and  on  their  Destruction, 
also  known  as  the  Ottawa  Treaty.  The  US  has  not  signed  off  on  the  illegal 
use  of  land  mines — indicating  willingness  to  use  non-discriminatory 
methods  if  required.29  Legal  mantraps  represent  a  class  of  autonomous 
weapons  employed  with  a  bit  more  discrimination 

Mantraps,  also  known  as  booby  traps,  defend  property  and 
personnel.  Some  states  within  the  US  have  recognized  legal  uses  of 
mantraps  as  a  means  of  home  defense  of  occupied  property.30  Restricting 
the  use  of  legal  mantraps  to  occupied  residences  requires  a  minimal  level 
of  discrimination  associated  with  perimeter  defense  of  an  occupied  area. 
Restricting  mantraps  to  personal  property  with  warning  signs  passively 


27  Canadian  Foreign  Affairs  and  International  Trade  Organization,  “When  and  Why  have  Land  Mines  Been 
Used,”  (March  16,  2012)  http://www.  international._gc.  ca/  mines/ background-apercu/use-mines- 
utilisation.aspx?lang=eng&view=d.  As  (accessed  on  April  1,  2012). 

28  Annup  Shah,  “Landmines,”  Global  Issues:  Social,  Political,  Economic,  and  Environmental  Issues  that 
Affect  Us  All  (November  27,  2009),  http://www.  globalissues._org/  article/  79/  landmines 

29  Shah,  “Landmines,”  Global  Issues. 

30  Case  Briefs,  “Katko  v.  Briney,  197  N.W.2d  351,  1972  Iowa  Sup.,”  http://www.casebriefs.  com/  blog/ 
law/  torts/torts-keyed-to-dobbs/defenses-to-intentional-torts-privileges/katko-v-briney/  (accessed  on  May9, 
2012).  “The  value  of  human  life  and  limb  both  to  an  individual  and  as  a  matter  of  public  policy  outweighs 
the  potential  damage  to  property.  Thus,  while  a  defendant  may  use  reasonable  force  in  defense  of  her 
property,  he  has  no  right  to  willfully  and  intentionally  injure  a  trespasser  in  a  manner  that  may  result  in  loss 
of  life  or  great  bodily  injury.  The  only  exception  is  when  the  trespasser  is  committing  a  violent  felony  with 
the  potential  of  endangering  human  life.”  A  mantrap  cannot  be  used  to  protect  property  however  a  mantrap 
protecting  an  occupied  dwelling  protects  the  occupants  from  potential  endangerment  to  human  life. 
Additionally,  the  history  of  mantraps  and  spring  guns  is  covered  in  Miller  Christy,  “Man-Traps  and  Spring- 
Guns,”  Sports  Library,  41,  http://la84foundation.  org /  SportsLibrary/  Outing/ 
Volume_41/outLXI06/outLXI06n.pdf  (accessed  May  9,  2012),  730. 


addresses  the  identification  aspect  of  discrimination.  Potential  victims 
are  considered  warned  against  entering  private  property  therefore  any 
violations  and  triggering  of  the  mantrap  is  limited  to  persons  violating 
the  warnings. 

The  integration  of  autonomous  systems  into  society  challenges 
legal  norms  today.31  The  related  analogies  presented  above  offer  glimpses 
into  legal  standing  arguments.  Standing  must  be  addressed  prior  to 
arguing  the  discriminate  and  proportional  legality  of  the  ALE’s  actions. 
Without  precedence,  a  declarative  judgment  based  on  related  analogies  is 
required  to  determine  standing. 

The  mantrap  and  landmine  cases  demonstrate  the  legal  flexibility 
related  to  autonomous  discrimination  capability.  These  two  examples  pin 
one  end  of  the  discriminatory  capability  of  autonomous  systems.  They 
are  used  to  emphasize  the  distinctive  evaluation  criteria  required  for  legal 
standing,  separate  from  lethal  autonomous  discrimination  capability. 
Therefore,  the  analogies  presented  for  legal  standing  are  applicable  to 
standing  determination  for  both  non-lethal  and  lethal  autonomous 
machines. 


31  Sven  A.  Beiker  and  M  Ryan  Calo,  “Legal  Aspects  of  Autonomous  Driving:  The  need  for  a  legal 
infrastructure  that  permits  autonomous  driving  in  public  to  maximize  safety  and  consumer  benefit.”  Center 
for  Automotive  Research  at  Stanford  (CARS)  and  the  Stanford  University  Center  or  Internet  and  Society 
(CIS),  Stanford  University  Press  (October  2010). 


Chapter  5:  Concluding  Remarks 

“Any  sufficiently  advanced  technology  is 
indistinguishable  from  magic.” 

-Arthur  C.  Clarke’s  Law 

Judge  Minos  entered  his  courtroom  and  set  eyes  on  the  ALE.  Am  I 
staring  at  a  machine  or  at  life  and  is  there  a  difference?  The  Judge 
wondered  if  Turing’s  test  had  been  passed. 1  After  reviewing  the  nagging 
questions  in  his  mind  concerning  the  ethical  justification  backed  by 
historical  case  studies  and  after  exploring  the  social  phenomenon  that 
led  to  the  current  trial,  Judge  Minos  decided  that  how  one  kills  is  less 
important  than  why  one  kills.  Context  matters. 

The  Ruling 

One  the  basis  of  moral  agency  and  unpredictable  contextual 
circumstances,  Judge  Minos  ruled  in  favor  of  legal  standing  for  the  ALE. 
Judge  Minos  was  challenged  with  the  question  of  “whether  the  robot  is 
correctly  seen  as  just  another  tool  or  is  it  something  more,  like  the 
technology  exemplified  by  the  guide  dog.”  The  Judge  did  not  view 
working  animals  on  the  same  level  as  humans  but  the  Judge  did 
understand  the  claim  that  “animals  have  their  own  wants,  desires  and 
states  of  well-being,  and  this  autonomy,  though  not  as  robust  as  that  of 
humans,  is  nonetheless  advanced  enough  to  give  the  dog  a  claim  for  both 
moral  rights  and  possibly  some  meager  moral  responsibilities  as  well.” 
The  machine  in  front  of  the  Judge  met  the  criteria  for  a  moral  agent. 
Through  demonstrated  discriminatory  ability,  the  Judge  deemed  the  ALE 
sufficiently  aware  of  potential  consequences  related  to  its  own  actions  to 
be  legally  accountable.  Moral  agent  status  combined  with  accountability 
formed  the  basis  of  Judge  Minos’s  declarative  ruling  for  legal  standing. 

1  Stanford  Encyclopedia  of  Philosophy,  “The  Turing  Test,”  http://plato.stanford.edu/entries/turing-test/ 
(First  published  Wed  Apr  9,  2003;  substantive  revision  Wed  Jan  26,  2011.  Accessed  May  9,  2012) 


As  with  natural  persons  as  well  as  corporate  persons, 
some  have  different  rights  than  others  do.  Same  rights 
don’t  apply  to  everyone  in  every  possible  situation. 

-  Retired  Supreme  Court  Justice 
John  Paul  Stevens 


Accountability 

Judge  Minos  also  recognized  the  Pandora’s  Box  opened  by  the 
declaration  of  legal  standing.  The  declaration  of  legal  standing  implies  a 
level  of  accountability  within  the  legal  system.  Autonomous  systems 
varied  greatly  in  levels  of  autonomy.  Not  all  autonomous  systems  are 
developed  enough  to  have  the  capability  to  discriminate  good  from  bad  or 
consider  the  consequences  of  acting.  Judge  Minos  predicted  a  varying 
level  of  accountability,  similar  to  those  used  for  cases  involving  children, 
as  the  legal  framework  for  autonomous  systems.  Judge  Minos  was  also 
concerned  with  the  ability  of  the  legal  system  to  impose  corrective 
measures  on  a  machine.  The  law  provides  a  framework  to  interpret 
wrongs  and  to  inflict  corrective  actions  or  punishment.  Is  it  possible  to 
take  physical,  emotional,  or  economic  repercussions  from  the  ALE  to 
execute  legal  corrective  actions? 

Punishment 

Punishment  falls  in  one  of  three  lines:  stalled  life  through 
incarceration,  economic  compensation,  or  loss  of  life  through  the  death 
penalty.2  All  three  punishments  have  parallels  for  an  autonomous 
system.  Incarceration  of  an  autonomous  machine  would  not  require  a 
physical  prison.  Assuming  autonomous  machines  evolve  through 
iterations  of  genetic  type  algorithms  and  interplay  with  advancing  neural 


2 

The  author  recognizes  the  distinction  between  criminal  punishment  and  civil  liability.  Since  the  ALE  is 
both  a  machine,  subject  to  destruction  by  court  order,  and  a  quasi  person,  subject  to  civil  liability,  both  civil 
liability  and  criminal  punishment  have  been  included  in  this  discussion. 


networks,  prohibiting  the  code  to  advance  for  a  set  number  of  cycles  is 
one  concept  for  incarceration. 

It  is  also  possible  to  invasion  enacting  economic  compensation 
against  autonomous  systems.  Assuming  the  autonomous  system  serves 
a  company,  a  government,  or  other  employer,  the  employers  may  opt  to 
maintain  use  of  the  autonomous  system  by  estimating  a  wage  based  on 
benefits  provided  by  the  ALE.  Wage  garnishment  is  then  an  option  for 
compensation.  Economic  compensations  are  also  possible  through 
assigned  labor  to  the  party  of  interest.  A  third  option  for  economic 
compensation  arises  from  selling  or  scrapping  the  material  and 
intellectual  code  of  the  autonomous  system.  Each  of  the  economic 
considerations  follows  historical  paths  used  for  indentured  humans 
unable  to  economically  satisfy  the  debt. 

To  consider  the  death  penalty  applicable  to  autonomous  machines, 
an  aspect  of  the  code  must  be  considered  as  a  form  of  life.  Eradication  of 
code  unique  to  the  autonomous  system  in  question  is  a  form  of  partial 
execution.  The  uniqueness  of  the  machine  is  terminated.  This 
punishment  allows  for  the  possibility  of  regrowth  from  a  generic  code  or 
complete  elimination.  This  same  methodology  applies  to  a  line  of 
autonomous  machines.  Depending  on  the  depth  of  the  punishment, 
destruction  of  the  core  code  across  all  linked  autonomous  machines 
would  punish  the  core  and  not  the  uniqueness  of  one  machine.  A  third 
form  of  death  punishment  may  take  the  form  of  destroying  the  body  but 
not  the  thinking  aspect  of  the  autonomous  machine.  In  this  case  a 
combination  of  incarceration  (separation  from  physical  form) ,  economic 
compensation  (one  reason  to  maintain  the  thinking  code  may  be  to 
require  work  to  compensate  damaged  parties) ,  and  a  form  of  death  by 
limiting  physical  interaction. 


The  Child  Analogy  and  Sliding  Persons  Rights 


The  evolution  of  autonomous  machines  is  an  underlying 
assumption  in  this  paper.  It  is  also  assumed  that  thinking  requires 
adaptability  and  thus,  for  machines,  computer  evolution.  The 
evolutionary  state  of  an  autonomous  system  challenges  the  legal  system. 
A  maturing  child  model  also  challenges  the  legal  system  and  is  thus  a 
useful  analogy. 

Accountability  for  an  autonomous  thinking  and  maturing  machine 
is  analogous  to  a  thinking  and  maturing  child.  Increased  legal  weight 
occurs  with  self-awareness  and  self-control  (rights  vary  with  age  and 
mental  capacity).  A  child,  whose  parents  are  divorcing,  has  varying  levels 
of  influence  concerning  custody  often  based  on  maturity.  The  right  for 
US  citizens  to  vote  occurs  at  the  maturity  age  of  18.  At  the  federal  level, 
some  US  rights  are  restricted  based  on  demonstrated  lack  of  self-control 
or  as  part  of  legal  punishment.  Forty-eight  states  within  the  US  “ban 
voting  by  felons  in  prison  or  on  supervision”  regardless  of  age.3 


3  Jonathan  Martin,  “Washington  state  felons  should  have  voting  rights,  federal  court  rules,”  Seattle  Times 
(January  5,  2012),  http://seattletimes.  nwsource.com/  html/localnews/  2010708869_felons06m.html 
(accessed  on  April  25,  2012). 


Conclusion 


The  intent  of  the  AFMC  question  was  to  “think  through  some  of  the 
ethics  and  trust  issues”  of  autonomous  lethal  system  before  research 
and  development  funds  are  heavily  invested. 1  The  proposed  Machine  on 
Trial  case  scratches  the  surface  of  these  issues  by  examining  one 
possible  future.  The  ALE  case  is  significant,  though  not  unique,  since  it 
concerns  accountability  driven  from  the  responsibility  and  authority  to 
take  life.  An  underlying  assumption  of  the  Machine  on  Trial  case  is  that 
the  legal  standing  trial  is  a  significant  and  discrete  legal  jump.  It  does 
not  account  for  probable  incremental  changes  to  the  law  concerning 
autonomous  systems.  Questions  concerning  legal  accountability  of 
autonomous  machines  are  rising  across  several  domains.  Basic 
autonomous  machines  exit  today.  They  sweep  floors,  manage  the  timing 
of  assembly  lines,  and  major  automobile  manufacturers  have 
demonstrated  levels  of  autonomous  vehicles. 

The  Machine  on  Trial  Case  challenges  the  root  assumptions  of  legal 
standing.  Weather  determined  or  socially  constructed,  technological 
advances  allow  for  the  possibility  of  machines  existing  with  significant 
autonomy  to  question  the  legal  foundations  and  the  basis  of  trust  within 
our  society.  Autonomous  systems  will  not  challenge  current  concepts  of 
reliability  or  ethics  related  to  justified  reasons  to  go  to  war  and  actions 
within  war.  The  reliability  of  autonomous  systems  is  a  quantifiable  value. 
For  a  given  probability,  reliability  sets  a  minimum  threshold  for 
prediction.  Once  quantified,  systems  exit  to  interpret  reliability  as  it 
relates  to  success  rate  and  predicted  behavior.  Reliability  should  not 
challenge  autonomous  systems. 

The  quantifiable  manner  of  reliability  does  not  pertain  to  questions 
of  trustworthiness.  Social  norms  are  one  area  of  influence  related  to 

1  Kevin  Stamey,  Director  of  Engineering  AFPEO/ISR  &  SOF,  email  “Re:  Regarding  Autonomous  Lethal 
Engagements,”  (August  8,  201 1). 


trust.  Shifting  moral  hazards  also  indicate  shifts  of  trust.  A  moral  hazard 
occurs  when  one  moral  agent,  with  assumed  responsibilities,  trusts  that 
another  moral  agent  has  assumed  risk  on  behalf  of  the  first  agent.  The 
impression  of  trust  creates  a  situation  where  the  first  moral  agent  may 
elect  to  change  behavior  due  to  a  perceived  shift  of  accountability. 

US  law  is  a  fluid  framework  intended  to  adjust  as  society  morphs. 
An  assumption  in  this  thesis  is  that  ALEs  will  not  change  international 
relations.  While  accepting  that  the  technology  may  allow  for  greater 
willingness  to  go  war,  it  is  assumed  that  ALEs  will  not  decide  when  one 
state  goes  to  war  with  another.  Case  studies  presented  in  the  thesis 
suggested  that  how  societies  kill  (direct  or  indirect)  is  less  critical  in  a 
legal  sense  than  why  societies  kill.  These  case  studies  also  suggest  that 
the  current  legal  framework,  based  on  acceptable  levels  of  discrimination 
and  proportionality,  are  sufficient  concerning  autonomous  systems.  The 
legal  crux  to  the  AFMC  question  centers  on  accountability.  Assuming 
autonomous  systems  are  inherently  different  from  a  tool  such  as  a 
hammer,  the  question  of  accountability  must  consider  the  autonomous 
system.  Legal  accountability  requires  legal  standing. 


Appendix  A:  Acronyms 


DARPA  -  Defense  Advanced  Research  Projects  Agency 

RPA  -  Remotely  Piloted  Aircraft 

UA  -  Unmanned  Aircraft 

UAS  -  Unmanned  Aircraft  System 

USAF  -  United  States  Air  Force 


Appendix  B:  Definitions 


autonomous  operation  —  In  air  defense,  the  mode  of  operation  assumed  by  a  unit  after 
it  has  lost  all  communications  with  higher  echelons.  The  unit  commander  assumes  full 
responsibility  for  control  of  weapons  and  engagement  of  hostile  targets.  (JP  1-02)  as 
stated  in  JP  1-02,  page  29. 

unmanned  aircraft  —  An  aircraft  or  balloon  that  does  not  carry  a  human  operator  and  is 
capable  of  flight  under  remote  control  or  autonomous  programming.  Also  called  UA.  (JP 
3-52)  as  stated  in  JP  1-02,  page  359. 

unmanned  aircraft  system  —  That  system  whose  components  include  the  necessary 
equipment,  network,  and  personnel  to  control  an  unmanned  aircraft.  Also  called  UAS.  (JP 
3-52)  as  stated  in  JP  1-02,  page  359. 
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